存儲(chǔ)網(wǎng)絡(luò)【免費(fèi)下載鏈接】glusterfsGluster Filesystem : Build your distributed storage in minutes項(xiàng)目地址https://gitcode.com/gh_mirrors/gl/glusterfs點(diǎn)擊查看免費(fèi)下載Statedump 是 GlusterFS 進(jìn)程在運(yùn)行時(shí)導(dǎo)出的內(nèi)部數(shù)據(jù)結(jié)構(gòu)快照文件覆蓋活動(dòng) inode、fd、內(nèi)存池mempool、iobuf、各 xlator 的數(shù)據(jù)結(jié)構(gòu)內(nèi)存分配統(tǒng)計(jì)等關(guān)鍵狀態(tài)。本文以 doc/debugging/statedump.md 為主線結(jié)合倉(cāng)庫(kù)源碼libglusterfs/src/statedump.c、cli/src/cli-cmd-volume.c 等深入講解如何對(duì) glusterd/glusterfs/glusterfsd 進(jìn)程生成 statedump、如何逐段解讀每一類輸出以及如何利用它定位內(nèi)存泄漏和由 frame 丟失引起的掛起問(wèn)題。什么是 StatedumpStatedump 是 GlusterFS 進(jìn)程生成的、記錄其內(nèi)部各種數(shù)據(jù)結(jié)構(gòu)狀態(tài)的文本文件。文件中可能包含當(dāng)前活動(dòng)的 inode、fd文件描述符內(nèi)存池mempool、iobuf 池的狀態(tài)每個(gè) xlator 各類數(shù)據(jù)結(jié)構(gòu)的內(nèi)存分配統(tǒng)計(jì)掛起的 call-stack / call-frame用于排查 fop 掛起fuse 層的操作歷史、xlator 私有配置、inode table 狀態(tài)等在 libglusterfs/src/statedump.c 中g(shù)f_proc_dump_info()是生成 statedump 的核心入口它通過(guò)gf_proc_dump_add_section()/gf_proc_dump_write()兩個(gè)工具函數(shù)以[section]與keyvalue的鍵值對(duì)格式逐段寫入 dump 文件見(jiàn) libglusterfs/src/statedump.c。也就是說(shuō)statedump 的所有區(qū)塊本質(zhì)上都來(lái)自各模塊主動(dòng)注冊(cè)的 dump 回調(diào)通過(guò)統(tǒng)一的鍵值文本格式輸出。如何生成 Statedump1. 確認(rèn) statedump 目錄使用gluster --print-statedumpdir命令查詢 statedump 文件的默認(rèn)存放目錄gluster --print-statedumpdir在源碼中該命令直接打印DEFAULT_VAR_RUN_DIRECTORY見(jiàn) cli/src/cli.c通常對(duì)應(yīng)/var/run/gluster一類路徑。如果該目錄在當(dāng)前安裝類型下尚不存在請(qǐng)先手動(dòng)創(chuàng)建。下文將該目錄統(tǒng)稱為statedump-directory。2. 使用 SIGUSR1 信號(hào)觸發(fā)對(duì)任意 GlusterFS 進(jìn)程glusterd / glusterfs / glusterfsd發(fā)送 SIGUSR1 即可觸發(fā)kill -USR1 pid-of-gluster-process從源碼看gf_proc_dump_info()就運(yùn)行在信號(hào)處理上下文gf_msg_trace(dump, 0, received statedump request (sig:USR1))見(jiàn) libglusterfs/src/statedump.c并受gf_proc_dump_mutex保護(hù)防止多次觸發(fā)相互干擾。值得注意的是statedump 落在信號(hào)處理器內(nèi)執(zhí)行因此實(shí)現(xiàn)中特意避免在該路徑上使用可能阻塞鎖的gf_log見(jiàn) libglusterfs/src/statedump.c。3. 針對(duì) brick / NFS / quotad 進(jìn)程Gluster 提供了專門的 CLI 命令來(lái)為不同類型進(jìn)程生成 statedumpgluster volume statedump volname gluster volume statedump volname nfs gluster volume statedump volname quotad其中第一條為brick 進(jìn)程brick 模式第二條為NFS 服務(wù)器進(jìn)程nfs 模式第三條為quotad 進(jìn)程quotad 模式。該命令的 CLI 側(cè)實(shí)現(xiàn)為cli_cmd_volume_statedump_cbk()見(jiàn) cli/src/cli-cmd-volume.c其完整的命令語(yǔ)法在命令注冊(cè)表中定義如下見(jiàn) cli/src/cli-cmd-volume.cvolume statedump VOLNAME [[nfs|quotad] [all|mem|iobuf|callpool|priv|fd|inode|history]... | [client hostname:process-id]]也就是說(shuō)除nfs、quotad外還支持all、mem、iobuf、callpool、priv、fd、inode、history等細(xì)分選項(xiàng)inodectx、fdctx亦被cli_cmd_validate_dumpoption()接受完整選項(xiàng)表見(jiàn) cli/src/cli-cmd-parser.c。這些選項(xiàng)決定了 dump 文件中包含哪些區(qū)塊對(duì)應(yīng)源碼中的gf_dump_options_t結(jié)構(gòu)libglusterfs/src/glusterfs/statedump.h。4. 針對(duì)使用 libgfapi 的應(yīng)用進(jìn)程對(duì)于通過(guò) libgfapi 接入 Gluster 的應(yīng)用如 smbd、libvirtd不能直接使用SIGUSR1——這類進(jìn)程可能早已把SIGUSR1用于其他用途。此時(shí)可在 Gluster 集群中該 libgfapi 應(yīng)用所連接的任一節(jié)點(diǎn)上執(zhí)行g(shù)luster volume statedump volname client hostname:process idCLI 解析器會(huì)校驗(yàn)hostname:pid格式其中 IP 必須是合法地址、pid 必須為正整數(shù)見(jiàn) cli/src/cli-cmd-parser.c。5. 生成文件的命名規(guī)則brick 進(jìn)程文件名為hyphenated-brick-path.pid.dump.timestampbrick 路徑中的/會(huì)替換為-見(jiàn) libglusterfs/src/statedump.c其他所有進(jìn)程文件名為glusterdump.pid.dump.timestamp。文件最終路徑的構(gòu)造邏輯見(jiàn) libglusterfs/src/statedump.c依次取glusterdump.options文件中的path配置 → 進(jìn)程自身的ctx-statedump_path→ 默認(rèn)目錄DEFAULT_VAR_RUN_DIRECTORY。寫入時(shí)先通過(guò)mkstemp創(chuàng)建臨時(shí)文件dumpXXXXXX全部寫完后再rename為最終文件名保證 dump 文件內(nèi)容完整見(jiàn) libglusterfs/src/statedump.c。對(duì)使用 libgfapi 的應(yīng)用進(jìn)程dump 文件同樣位于statedump-directory命名為glusterdump.pid.dump.timestamp。由于一個(gè)應(yīng)用可能多次調(diào)用glfs_init()訪問(wèn)多個(gè)卷同一個(gè) pid 可能對(duì)應(yīng)多個(gè)這樣的文件。6. 通過(guò) glusterdump.options 控制 dump 內(nèi)容直接發(fā)送SIGUSR1時(shí)沒(méi)有命令行選項(xiàng)機(jī)制。為了讓 CLI 命令和 SIGUSR1 兩種方式都能配置 dump 內(nèi)容進(jìn)程會(huì)優(yōu)先讀取statedump-directory/glusterdump.options全局配置其次讀取statedump-directory/glusterdump.pid.options按進(jìn)程配置具體邏輯見(jiàn) libglusterfs/src/statedump.c兩個(gè)文件都不存在時(shí)默認(rèn)開(kāi)啟全部選項(xiàng)gf_proc_dump_enable_all_options()存在文件時(shí)先關(guān)閉全部選項(xiàng)再按文件中keyvalue逐行解析value以yes開(kāi)頭即為開(kāi)啟如memyes、iobufyes若解析后所有選項(xiàng)都被禁用則回退到默認(rèn)選項(xiàng)mem與callpoolgf_proc_dump_enable_default_options()。如何閱讀 Statedump下面按區(qū)塊逐一解讀各類輸出的含義。文件首尾時(shí)間戳dump 文件第一行和最后一行分別記錄了開(kāi)始與結(jié)束寫入 statedump 的時(shí)間格式為DUMP-START-TIME/DUMP-END-TIME時(shí)區(qū)為UTC實(shí)現(xiàn)見(jiàn) libglusterfs/src/statedump.c。兩端時(shí)間差可粗略衡量 dump 寫入耗時(shí)。Mallinfoglibc 堆內(nèi)存統(tǒng)計(jì)mallinfo區(qū)塊來(lái)自 glibc 的mallinfo()/mallinfo2()源碼根據(jù)HAVE_MALLINFO2宏選擇調(diào)用見(jiàn) libglusterfs/src/statedump.c反映進(jìn)程堆的總體分配狀況[mallinfo] mallinfo_arena100020224 /* Non-mmapped space allocated (bytes) */ mallinfo_ordblks69467 /* Number of free chunks */ mallinfo_smblks449 /* Number of free fastbin blocks */ mallinfo_hblks13 /* Number of mmapped regions */ mallinfo_hblkhd20144128 /* Space allocated in mmapped regions (bytes) */ mallinfo_usmblks0 /* Maximum total allocated space (bytes) */ mallinfo_fsmblks39264 /* Space in freed fastbin blocks (bytes) */ mallinfo_uordblks96710112 /* Total allocated space (bytes) */ mallinfo_fordblks3310112 /* Total free space (bytes) */ mallinfo_keepcost133712 /* Top-most, releasable space (bytes) */各字段的詳細(xì)語(yǔ)義請(qǐng)參考man mallinfo。簡(jiǎn)單來(lái)說(shuō)mallinfo_uordblks是已分配空間mallinfo_fordblks是空閑空間mallinfo_arena是非 mmap 區(qū)域的分配量mallinfo_hblkhd是 mmap 區(qū)域的分配量。它是判斷進(jìn)程整體內(nèi)存增長(zhǎng)趨勢(shì)的起點(diǎn)但要定位具體是哪類數(shù)據(jù)結(jié)構(gòu)在增長(zhǎng)還需要結(jié)合下面兩個(gè)區(qū)塊。數(shù)據(jù)結(jié)構(gòu)分配統(tǒng)計(jì)Data structure allocation stats對(duì)調(diào)用圖call-graph中加載的每個(gè) xlatorstatedump 會(huì)按數(shù)據(jù)結(jié)構(gòu)類型打印內(nèi)存記賬memory accounting信息。先給出匯總頭[global.glusterfs - Memory usage] #[global.xlator-name - Memory usage] num_types119 #It shows the number of data types it is using其中num_types是該 xlator 使用的數(shù)據(jù)類型數(shù)量。隨后對(duì)每一種有活動(dòng)分配的數(shù)據(jù)類型打印明細(xì)格式如下[global.glusterfs - usage-type gf_common_mt_gf_timer_t memusage] #[global.xlator-name - usage-type tag associated with the>[mempool] #Section name ---------- pool-namefuse:fd_t #pool-namexlator-name:data-type hot-count1 #number of mempool elements that are in active use. i.e. for this pool it is the number of fd_t s in active use. cold-count1023 #number of mempool elements that are not in use. If a new allocation is required it will be served from here until all the elements in the pool are in use i.e. cold-count becomes 0. padded_sizeof108 #Each mempool element is padded with a doubly-linked-list ptr of mempool is-in-use info to operate the pool of elements, this size is the element-size after padding pool-misses0 #Number of times the element had to be allocated from heap because all elements from the pool are in active use. alloc-count314 #Number of times this type of data is allocated through out the life of this process. This may include pool-misses as well. max-alloc3 #Maximum number of elements from the pool in active use at any point in the life of the process. This does *not* include pool-misses. cur-stdalloc0 #Denotes the number of allocations made from heap once cold-count reaches 0, that are yet to be released via mem_put(). max-stdalloc0 #Maximum number of allocations from heap that are in active use at any point in the life of the process.各字段含義字段含義pool-name池名稱格式為xlator-name:data-typehot-count池中處于活動(dòng)使用狀態(tài)的元素?cái)?shù)如該池即當(dāng)前活動(dòng)的fd_t數(shù)cold-count池中空閑未使用元素?cái)?shù)新分配優(yōu)先從這里取直到池元素全部被占用cold-count 為 0padded_sizeof每個(gè)元素經(jīng)過(guò)填充后的實(shí)際大小含雙向鏈表、池指針、是否在用標(biāo)記等池管理開(kāi)銷pool-misses因池元素全部在用而從堆分配的次數(shù)alloc-count進(jìn)程生命周期內(nèi)該類型數(shù)據(jù)的總分配次數(shù)含 pool-missesmax-alloc進(jìn)程生命周期內(nèi)池元素活動(dòng)使用數(shù)的歷史峰值不含pool-missescur-stdalloccold-count 歸零后從堆分配、且尚未通過(guò)mem_put()歸還的分配數(shù)max-stdalloc任意時(shí)刻從堆分配且處于活動(dòng)使用的最大數(shù)量cold-count長(zhǎng)期為 0 且pool-misses持續(xù)增長(zhǎng)通常意味著內(nèi)存池被打穿、系統(tǒng)頻繁走堆分配cur-stdalloc持續(xù)增長(zhǎng)則說(shuō)明有分配沒(méi)有歸還是內(nèi)存泄漏的強(qiáng)信號(hào)。IobufsI/O 緩沖池iobuf 是 GlusterFS 為 I/O 數(shù)據(jù)準(zhǔn)備的緩沖池其全局統(tǒng)計(jì)格式如下[iobuf.global] iobuf_pool0x1f0d970 #The memory pool for iobufs iobuf_pool.default_page_size131072 #The default size of iobuf (if no iobuf size is specified the default size is allocated) #iobuf_arena: One arena represents a group of iobufs of a particular size iobuf_pool.arena_size12976128 # The initial size of the iobuf pool (doesnt include the stdallocd memory or the newly added arenas) iobuf_pool.arena_cnt8 #Total number of arenas in the pool iobuf_pool.request_misses0 #The number of iobufs that were stdallocd (as they exceeded the default max page size provided by iobuf_pool).iobuf_pool.request_misses表示因?yàn)檎?qǐng)求尺寸超過(guò)池默認(rèn)最大頁(yè)大小而走 stdalloc 的次數(shù)。這些字段與 libglusterfs/src/iobuf.c 中iobuf_stats_dump()的輸出一一對(duì)應(yīng)。iobuf 池中存在3 種 arena 列表Arena 列表池創(chuàng)建時(shí)分配的 arena以及正在使用active_cnt ! 0的 arenaPurge 列表可以被回收的 arena無(wú)活動(dòng) iobufactive_cnt 0Filled 列表沒(méi)有空閑 iobuf 的 arena。每個(gè) arena 的明細(xì)如下[purge.1] #purge.S.No. purge.1.mem_base0x7fc47b35f000 #The address of the arena structure purge.1.active_cnt0 #The number of iobufs active in that arena purge.1.passive_cnt1024 #The number of unused iobufs in the arena purge.1.alloc_cnt22853 #Total allocs in this pool(number of times the iobuf was allocated from this arena) purge.1.max_active7 #Max active iobufs from this arena, at any point in the life of this process. purge.1.page_size128 #Size of all the iobufs in this arena. [arena.5] #arena.S.No. arena.5.mem_base0x7fc47af1f000 arena.5.active_cnt0 arena.5.passive_cnt64 arena.5.alloc_cnt0 arena.5.max_active0 arena.5.page_size32768如果某個(gè) arena 的active_cnt非零statedump 還會(huì)額外打印該 arena 的活動(dòng) iobuf 列表[arena.6.active_iobuf.1] #arena.S.No.active_iobuf.iobuf.S.No. arena.6.active_iobuf.1.ref1 #refcount of the iobuf arena.6.active_iobuf.1.ptr0x7fdb921a9000 #address of the iobuf [arena.6.active_iobuf.2] arena.6.active_iobuf.2.ref1 arena.6.active_iobuf.2.ptr0x7fdb92189000在任意時(shí)刻如果存在大量filled arena無(wú)空閑 iobuf則可能是iobuf 泄漏的跡象值得結(jié)合活動(dòng) iobuf 列表與調(diào)用棧進(jìn)一步分析。Call stack調(diào)用棧GlusterFS 收到的所有 fop 都通過(guò) call-stack 處理。call-stack 中包含執(zhí)行該 fop 的進(jìn)程的 uid/gid/pid 等信息每個(gè) call-stack 由多個(gè) call-frame 組成——每個(gè) frame 對(duì)應(yīng)處理該 fop 路徑上的一個(gè) xlator。格式如下[global.callpool.stack.3] #global.callpool.stack.Serial-Number stack0x7fc47a44bbe0 #Stack address uid0 #Uid of the process which is executing the fop gid0 #Gid of the process which is executing the fop pid6223 #Pid of the process which is executing the fop unique2778 #Xlators like afr do copy_frame and perform the operation in a different stack, this id is useful to find out the stacks that are inter-related because of copy-frame lk-owner0000000000000000 #Some of the fuse fops have lk-owner. opLOOKUP #Fop type1 #Type of the op i.e. FOP/MGMT-OP cnt9 #Number of frames in this stack.其中unique尤其值得注意像 afr 這類 xlator 會(huì)通過(guò)copy_frame在新棧中執(zhí)行操作unique用于關(guān)聯(lián)因 copy-frame 而相互關(guān)聯(lián)的多個(gè)棧。Call-frame調(diào)用幀每個(gè) frame 記錄其所屬 xlator、從哪個(gè)函數(shù) wind、向哪個(gè)函數(shù) unwind以及是否已完成 unwind。當(dāng)系統(tǒng)出現(xiàn)掛起時(shí)取一個(gè) statedump找到最后一個(gè)尚未 unwind 的 xlator就能定位掛起發(fā)生的層[global.callpool.stack.3.frame.2]#global.callpool.stack.stack-serial-number.frame.frame-serial-number frame0x7fc47a611dbc #Frame address ref_count0 #Incremented at the time of wind and decremented at the time of unwind. translatorr2-client-1 #Xlator this frame belongs to complete0 #if this value is 1 that means this frame is already unwound. 0 if it is yet to unwind. parentr2-replicate-0 #Parent xlator of this frame wind_fromafr_lookup #Parent xlator function from which the wind happened wind_topriv-children[i]-fops-lookup unwind_toafr_lookup_cbk #Parent xlator function to which unwind happened關(guān)鍵字段字段含義ref_countwind 時(shí)遞增、unwind 時(shí)遞減的引用計(jì)數(shù)translator該 frame 所屬的 xlatorcomplete1 表示該 frame 已 unwind0 表示尚未 unwindparent該 frame 的父 xlatorwind_from父 xlator 中發(fā)起 wind 的函數(shù)wind_towind 目標(biāo)下一層 xlator 的 fopunwind_to父 xlator 中接收 unwind 的回調(diào)函數(shù)如果棧中靠前接近客戶端的 framecomplete1而靠后的 framecomplete0說(shuō)明調(diào)用在某個(gè) xlator 處“丟失”了具體排查流程見(jiàn)文末 FAQ。Fuse 操作歷史History of operations in Fusefuse xlator 會(huì)維護(hù)在 fuse 層發(fā)生過(guò)的操作歷史輸出格式如下[xlator.mount.fuse.history] TIME2014-07-09 16:44:57.523364 message[0] fuse_release: RELEASE(): 4590:, fd: 0x1fef0d8, gfid: 3afb4968-5100-478d-91e9-76264e634c9f TIME2014-07-09 16:44:57.523373 message[0] send_fuse_err: Sending Success for operation 18 on inode 3afb4968-5100-478d-91e9-76264e634c9f TIME2014-07-09 16:44:57.523394 message[0] fuse_getattr_resume: 4591, STAT, path: (/iozone.tmp), gfid: (3afb4968-5100-478d-91e9-76264e634c9f)歷史記錄由 fuse xlator 的fuse_history_dump()回調(diào)輸出且受event-history選項(xiàng)控制見(jiàn) xlators/mount/fuse/src/fuse-bridge.c。掛起發(fā)生時(shí)這段歷史能幫你判斷最后一個(gè)成功處理的操作是什么。Xlator 配置Xlator configuration每個(gè) xlator 的私有配置priv也會(huì)被 dump格式如下[cluster/replicate.r2-replicate-0] #Xlator type, name information child_count2 #Number of children to the xlator #Xlator specific configuration below child_up[0]1 pending_key[0]trusted.afr.r2-client-0 child_up[1]1 pending_key[1]trusted.afr.r2-client-1 data_self_healon metadata_self_heal1 entry_self_heal1 data_change_log1 metadata_change_log1 entry-change_log1 read_child1 favorite_child-1 wait_count1區(qū)塊標(biāo)題格式為[xlator-type.xlator-name]隨后是child_count與 xlator 特有的配置項(xiàng)此處為 replicate/afr 的 child 狀態(tài)、self-heal 開(kāi)關(guān)、pending key 等。這部分通過(guò)各 xlator 注冊(cè)的dumpops-priv回調(diào)輸出見(jiàn) libglusterfs/src/statedump.c。Graph / inode table當(dāng)前活動(dòng) graph 與 inode table 的統(tǒng)計(jì)如下[active graph - 1] conn.1.bound_xl./data/brick01a/homegfs.hashsize14057 conn.1.bound_xl./data/brick01a/homegfs.name/data/brick01a/homegfs/inode conn.1.bound_xl./data/brick01a/homegfs.lru_limit16384 #Least recently used size limit conn.1.bound_xl./data/brick01a/homegfs.active_size690 #Number of inodes undergoing some kind of fop to be precise on which there is at least one ref. conn.1.bound_xl./data/brick01a/homegfs.lru_size183 #Number of inodes present in lru list conn.1.bound_xl./data/brick01a/homegfs.purge_size0 #Number of inodes present in purge list關(guān)鍵統(tǒng)計(jì)active_size正在參與某種 fop、至少持有一個(gè)引用的 inode 數(shù)lru_size處于 LRU 列表中的 inode 數(shù)purge_size處于 purge 列表中的 inode 數(shù)lru_limitLRU 大小的上限。若active_size異常偏高且持續(xù)不降說(shuō)明 inode 引用未被正確釋放。Inode 明細(xì)inode table 中的每個(gè) inode 也會(huì)被打印格式如下[conn.1.bound_xl./data/brick01a/homegfs.active.324] #324th inode in active inode list gfide6d337cf-97eb-44b3-9492-379ba3f6ad42 #Gfid of the inode nlookup13 #Number of times lookups happened from the client or from fuse kernel fd-count4 #Number of fds opened on the inode ref11 #Number of refs taken on the inode ia_type1 #Type of the inode. This should be changed to some string :-( Ref by xl:.patchy-md-cache11 #Further this there will be a list of xlators, and the ref count taken by each of them on this inode at the time of statedump [conn.1.bound_xl./data/brick01a/homegfs.lru.1] #1st inode in lru list. Note that ref count is zero for these inodes. gfid5114574e-69bc-412b-9e52-f13ff087c6fc nlookup5 fd-count0 ref0 ia_type2 Ref by xl:.fuse1 Ref by xl:.patchy-client-0-1要點(diǎn)active 列表中的 inoderef非零lru 列表中的 inode ref 為 0如上面的ref0所示Ref by xl:xlator名ref數(shù)逐條列出每個(gè) xlator 對(duì)當(dāng)前 inode 持有的引用數(shù)用于判斷是誰(shuí)“拖住”了 inode 沒(méi)有釋放fd-count為 on inode 上打開(kāi)的文件描述符數(shù)長(zhǎng)期不為 0 的 inode 可能與 fd 泄漏相關(guān)。Inode context含鎖信息每個(gè) xlator 還可能為 inode 保存上下文例如 locks xlator 的 inode 上下文會(huì)輸出鎖域與鎖明細(xì)[xlator.features.locks.homegfs-locks.inode] path/homegfs/users/dfrobins/gfstest/r4/SCRATCH/fort.5102 - path of the file mandatory0 inodelk-count5 #Number of inode locks lock-dump.domain.domainhomegfs-replicate-0:self-heal #Domain name where self-heals take locks to prevent more than one heal on the same file inodelk.inodelk0typeWRITE, whence0, start0, len0, pid 18446744073709551615, owner080b1ada117f0000, client0xb7fc30, connection-idcompute-30-029.com-3505-2014/06/29-14:46:12:477358-homegfs-client-0-0-1, granted at Sun Jun 29 11:01:00 2014 #Active lock information inodelk.inodelk1typeWRITE, whence0, start0, len0, pid 18446744073709551615, ownerc0cb091a277f0000, client0xad4f10, connection-idgfs01a.com-4080-2014/06/29-14:41:36:917768-homegfs-client-0-0-0, blocked at Sun Jun 29 11:04:44 2014 #Blocked lock information lock-dump.domain.domainhomegfs-replicate-0:metadata #Domain name where metadata operations take locks to maintain replication consistency lock-dump.domain.domainhomegfs-replicate-0 #Domain name where entry/data operations take locks to maintain replication consistency inodelk.inodelk0typeWRITE, whence0, start11141120, len131072, pid 18446744073709551615, owner080b1ada117f0000, client0xb7fc30, connection-idcompute-30-029.com-3505-2014/06/29-14:46:12:477358-homegfs-client-0-0-1, granted at Sun Jun 29 11:10:36 2014 #Active lock information鎖信息按鎖域domain分組self-heal、metadata、entry/data 等每個(gè)鎖條目標(biāo)注了類型ACTIVE/BLOCKED、鎖類型READ/WRITE、起始偏移與長(zhǎng)度、owner、client、連接 ID 以及加鎖/阻塞時(shí)間。當(dāng)懷疑鎖導(dǎo)致 I/O 掛起時(shí)這份數(shù)據(jù)可以直接給出“誰(shuí)在等誰(shuí)”的證據(jù)。實(shí)戰(zhàn) FAQ用 Statedump 排查典型問(wèn)題如何用 Statedump 調(diào)試內(nèi)存泄漏方法一利用內(nèi)存記賬memory accounting特性一個(gè)真實(shí)案例對(duì)應(yīng) Bugzilla 1120151在 replicate 卷中擦除一個(gè) brick 并觸發(fā)fullself-heal 后進(jìn)程內(nèi)存持續(xù)增長(zhǎng)。對(duì) self-heal 守護(hù)進(jìn)程執(zhí)行kill -USR1 pid-of-gluster-self-heal-daemon抓取 statedump然后grep -w num_allocs glusterdump.5225.dump.1405493251 num_allocs77078 num_allocs87070 num_allocs117376 .... grep hot-count glusterdump.5225.dump.1405493251 hot-count16384 hot-count16384 hot-count4095 ....在 statedump 文件中定位這些高數(shù)值對(duì)應(yīng)的數(shù)據(jù)類型 tag。該案例中g(shù)rep 結(jié)果顯示 replicate 下以下數(shù)據(jù)類型的分配異常偏高gf_common_mt_asprintfgf_common_mt_chargf_common_mt_mem_pool隨后排查 afr 源碼中帶gf_common_mt_chartag 的分配發(fā)現(xiàn)data-self-heal代碼路徑有一處內(nèi)存沒(méi)有釋放gf_common_mt_mem_pool則提示池內(nèi)存存在泄漏——replicate-0:dict_t、glusterfs:data_t、glusterfs:data_pair_t這些池的cold_count為 0 且分配過(guò)多。檢查 libglusterfs/src/dict.c 源碼后發(fā)現(xiàn)dict中的key使用gf_common_mt_char分配對(duì)應(yīng)第 2 個(gè) tagvalue 通過(guò)gf_asprintf創(chuàng)建內(nèi)部使用gf_common_mt_asprintf對(duì)應(yīng)第 1 個(gè) tag。繼續(xù)在 self-heal 代碼路徑中排查定位到一行代碼在變量已持有某個(gè)字典引用的情況下又用新字典覆蓋了它舊引用丟失。修復(fù)泄漏后在同一測(cè)試heal 包含 10,000 個(gè)文件的目錄層級(jí)中驗(yàn)證self-heal 守護(hù)進(jìn)程 statedump 里所有num_allocs不再增長(zhǎng)。排查思路小結(jié)num_allocs反映“當(dāng)前活動(dòng)分配數(shù)”若某 tag 的num_allocs隨測(cè)試推進(jìn)持續(xù)上升就說(shuō)明該類型數(shù)據(jù)結(jié)構(gòu)只增不減順著 tag 反查源碼中對(duì)應(yīng)分配點(diǎn)即可鎖定泄漏位置。方法二排查內(nèi)存池泄漏借助 -DDEBUG 構(gòu)建另一個(gè)案例對(duì)應(yīng) Bugzilla 1134221代碼分析發(fā)現(xiàn) name self-heal 過(guò)程中dict_t對(duì)象被反復(fù)創(chuàng)建而未被充分 unref。驗(yàn)證步驟如下用-DDEBUG編譯 glusterfs使cold count默認(rèn)為 0即內(nèi)存池默認(rèn)不預(yù)分配讓堆分配情況完全暴露在cur-stdalloc中在 plain replicate 卷上創(chuàng)建 100 個(gè)文件從其中一個(gè) brick 的后端刪除它們從掛載點(diǎn)對(duì)這些路徑觸發(fā) lookup測(cè)試前后分別對(duì) fuse 掛載進(jìn)程取 statedump。測(cè)試前的 statedumpfuse mount 進(jìn)程pool-nameglusterfs:dict_t hot-count0 cold-count0 padded_sizeof140 alloc-count33 max-alloc0 pool-misses33 cur-stdalloc14 max-stdalloc18測(cè)試后的 statedumppool-nameglusterfs:dict_t hot-count0 cold-count0 padded_sizeof140 alloc-count2841 max-alloc0 pool-misses2841 cur-stdalloc214 max-stdalloc220在cold count為 0 的前提下cur-stdalloc表示通過(guò)mem_get()從堆分配、尚未用mem_put()歸還的dict_t對(duì)象數(shù)。測(cè)試100 個(gè)文件的 name self-heal之后dict_t的cur-stdalloc從 14 漲到 214泄漏被證實(shí)。mempool 工作機(jī)制的更多細(xì)節(jié)可參考 datastructure-mem-pool.md。修復(fù)泄漏后再次以-DDEBUG編譯并重復(fù)相同步驟測(cè)試前后的結(jié)果如下——修復(fù)前pool-nameglusterfs:dict_t hot-count0 cold-count0 padded_sizeof140 alloc-count33 max-alloc0 pool-misses33 cur-stdalloc14 max-stdalloc18修復(fù)后測(cè)試后pool-nameglusterfs:dict_t hot-count0 cold-count0 padded_sizeof140 alloc-count2837 max-alloc0 pool-misses2837 cur-stdalloc14 max-stdalloc119cur-stdalloc在測(cè)試前后保持 14 不變分配峰值max-stdalloc也明顯回落說(shuō)明修復(fù)確實(shí)生效。排查思路小結(jié)cur-stdalloc是堆上“已分配未歸還”對(duì)象的實(shí)時(shí)計(jì)數(shù)測(cè)試前后對(duì)比該值即可判斷是否有對(duì)象被分配后從未釋放。如何用 Statedump 調(diào)試因 frame 丟失導(dǎo)致的掛起真實(shí)案例對(duì)應(yīng) Bugzilla 994959中掛起發(fā)生時(shí)對(duì)所有進(jìn)程抓取 statedump。mount 進(jìn)程的 statedump 中出現(xiàn)了如下調(diào)用棧[global.callpool.stack.1.frame.1] ref_count1 translatorfuse complete0 [global.callpool.stack.1.frame.2] ref_count0 translatorr2-client-1 complete1 ----- Client xlator completed the readdirp call and unwound to afr parentr2-replicate-0 wind_fromafr_do_readdir wind_tochildren[call_child]-fops-readdirp unwind_fromclient3_3_readdirp_cbk unwind_toafr_readdirp_cbk [global.callpool.stack.1.frame.3] ref_count0 translatorr2-replicate-0 complete0 ---- Afr xlator is not unwinding for some reason. parentr2-dht wind_fromdht_do_readdir wind_toxvol-fops-readdirp unwind_todht_readdirp_cbk [global.callpool.stack.1.frame.4] ref_count1 translatorr2-dht complete0 parentr2-io-cache wind_fromioc_readdirp wind_toFIRST_CHILD(this)-fops-readdirp unwind_toioc_readdirp_cbk [global.callpool.stack.1.frame.5] ref_count1 translatorr2-io-cache complete0 parentr2-quick-read wind_fromqr_readdirp wind_toFIRST_CHILD (this)-fops-readdirp unwind_toqr_readdirp_cbk分析要點(diǎn)frame.2r2-client-1complete1說(shuō)明 client xlator 已完成readdirp調(diào)用并 unwind 回 afr其unwind_toafr_readdirp_cbkframe.3r2-replicate-0complete0說(shuō)明afr xlator 沒(méi)有繼續(xù) unwind其下的 frame.4、frame.5dht、io-cache、quick-read自然全部處于complete0的掛起狀態(tài)。順著unwind_toafr_readdirp_cbk檢查 afr 源碼中該回調(diào)的實(shí)現(xiàn)發(fā)現(xiàn)當(dāng) fop 失敗時(shí)afr 沒(méi)有對(duì)棧執(zhí)行 unwind導(dǎo)致整個(gè)調(diào)用鏈懸掛。找到根因并修復(fù)后掛起消失。排查思路小結(jié)在棧中尋找“上方 frame 已 complete、但本 frame 仍未 complete”的斷層點(diǎn)——那個(gè) frame 所屬的 xlator 就是 fop 被卡住的地方。unwind_to字段直接指明了應(yīng)當(dāng)被調(diào)用卻沒(méi)有被調(diào)用的回調(diào)函數(shù)據(jù)此即可精確定位源碼位置。附相關(guān)源碼路徑速查關(guān)注點(diǎn)文件statedump 核心實(shí)現(xiàn)信號(hào)處理、文件生成、選項(xiàng)解析libglusterfs/src/statedump.cstatedump 選項(xiàng)結(jié)構(gòu)定義與對(duì)外 APIlibglusterfs/src/glusterfs/statedump.hiobuf 池狀態(tài) dumparena/purge/filled 列表libglusterfs/src/iobuf.cgluster volume statedumpCLI 命令回調(diào)cli/src/cli-cmd-volume.cstatedump 選項(xiàng)校驗(yàn)與client hostname:pid解析cli/src/cli-cmd-parser.cgluster --print-statedumpdir實(shí)現(xiàn)cli/src/cli.cfuse 操作歷史 dump 回調(diào)xlators/mount/fuse/src/fuse-bridge.cmempool 數(shù)據(jù)結(jié)構(gòu)設(shè)計(jì)說(shuō)明datastructure-mem-pool.md總結(jié)Statedump 是 GlusterFS 運(yùn)維與開(kāi)發(fā)調(diào)試中最高效的“現(xiàn)場(chǎng)快照”工具kill -USR1與gluster volume statedump兩條路徑覆蓋 glusterd、brick、NFS、quotad 以及 libgfapi 應(yīng)用進(jìn)程輸出中的 mallinfo、數(shù)據(jù)結(jié)構(gòu)分配統(tǒng)計(jì)、mempool、iobuf、call-stack/call-frame、fuse 歷史、鎖上下文等區(qū)塊分別服務(wù)于內(nèi)存泄漏定位、iobuf 泄漏判斷、fop 掛起歸因等典型故障場(chǎng)景。掌握“先抓 dump、再按區(qū)塊交叉印證、最后反查源碼”的方法論就能把一次莫名的內(nèi)存增長(zhǎng)或 I/O 掛起快速收斂到具體的 xlator、數(shù)據(jù)結(jié)構(gòu)和代碼路徑。贊分享存儲(chǔ)網(wǎng)絡(luò)【免費(fèi)下載鏈接】glusterfsGluster Filesystem : Build your distributed storage in minutes項(xiàng)目地址https://gitcode.com/gh_mirrors/gl/glusterfs點(diǎn)擊查看免費(fèi)下載相關(guān)推薦CodeForge內(nèi)存泄漏問(wèn)題排查方法CodeForge內(nèi)存泄漏問(wèn)題排查方法 引言為什么內(nèi)存泄漏是CodeForge的關(guān)鍵問(wèn)題 作為一款多語(yǔ)言代碼執(zhí)行器CodeForge面臨著獨(dú)特的內(nèi)存管理挑開(kāi)發(fā)工具代碼編輯器桌面應(yīng)用數(shù)據(jù)庫(kù)客戶端數(shù)據(jù)可視化MimicKit深度解析輕量級(jí)運(yùn)動(dòng)模仿框架如何徹底改變控制器訓(xùn)練MimicKit深度解析輕量級(jí)運(yùn)動(dòng)模仿框架如何徹底改變控制器訓(xùn)練 MimicKit是一款輕量級(jí)運(yùn)動(dòng)模仿框架專為訓(xùn)練控制器提供一套全面的運(yùn)動(dòng)模仿方法。該框架告別內(nèi)存噩夢(mèng)前端內(nèi)存泄漏完全排查指南告別內(nèi)存噩夢(mèng)前端內(nèi)存泄漏完全排查指南 你是否遇到過(guò)網(wǎng)頁(yè)越用越卡甚至崩潰的情況用戶投訴頁(yè)面操作卡頓卻找不到明確報(bào)錯(cuò)這很可能是內(nèi)存泄漏在作祟。本文將帶你從文檔技術(shù)博客教程上一篇完整指南How to Train Your GPT 詳解 LLM 推理加速與采樣參數(shù)設(shè)置KV Cache、Temperature、Top-k/Top-P 一次講透下一篇Laya-CoreML W8 調(diào)色板壓縮實(shí)戰(zhàn)K-means 權(quán)重量化帶來(lái)3.19倍能效的完整過(guò)程創(chuàng)作聲明：本文部分內(nèi)容由AI輔助生成（AIGC），僅供參考