redis-rdb-tools来解析分析reids dump文件及内存使用量-成长的对话

一. 前言

解析redis的dump.rdb文件，分析内存，以JSON格式导出数据。|
提供的功能有：
1. 生成内存报告
2. 转储文件到JSON
3. 使用标准的diff工具比较两个dump文件

Rdbtools是以python语言开发的。

二. 安装

2.1 前提条件
1. python2.4以上版本和 pip
2. redis-py可选，只运行在测试用例下

2.2 从PyPI安装(推荐)

# /usr/local/python/bin/easy_install pip
# /usr/local/python/bin/pip install rdbtools

1 2	# /usr/local/python/bin/easy_install pip # /usr/local/python/bin/pip install rdbtools

2.3 从源码包安装

# wget https://github.com/sripathikrishnan/redis-rdb-tools/archive/master.zip
# unzip master
# cd redis-rdb-tools-master/
# python setup.py install
Downloading/unpacking rdbtools
  Downloading rdbtools-0.1.5.tar.gz
  Running setup.py egg_info for package rdbtools

    warning: no files found matching 'README.textile'
Installing collected packages: rdbtools
  Running setup.py install for rdbtools

    warning: no files found matching 'README.textile'
    Installing redis-memory-for-key script to /usr/local/python/bin
    Installing redis-profiler script to /usr/local/python/bin
    Installing rdb script to /usr/local/python/bin
Successfully installed rdbtools
Cleaning up...

# wget https://github.com/sripathikrishnan/redis-rdb-tools/archive/master.zip

# unzip master

# cd redis-rdb-tools-master/

# python setup.py install

Downloading / unpacking rdbtools

Downloading rdbtools - 0.1.5.tar.gz

Running setup . py egg_info for package rdbtools

warning : no files found matching 'README.textile'

Installing collected packages : rdbtools

Running setup . py install for rdbtools

warning : no files found matching 'README.textile'

Installing redis - memory - for - key script to / usr / local / python / bin

Installing redis - profiler script to / usr / local / python / bin

Installing rdb script to / usr / local / python / bin

Successfully installed rdbtools

Cleaning up . . .

三. 转换dump文件到JSON

# /usr/local/python/bin/rdb --help
Usage: rdb [options] /path/to/dump.rdb
Example : rdb --command json -k "user.*" /var/redis/6379/dump.rdb

Options:
  -h, --help            show this help message and exit
  -c FILE, --command=FILE
                        要执行的命令json 或 diff
  -f FILE, --file=FILE  输出文件名
  -n DBS, --db=DBS      数据库ID。可以提供多个数据库。如果没有指定，包含所有数据库。
  -k KEYS, --key=KEYS   导出键。可以是正则表达式。
  -t TYPES, --type=TYPES
                        数据类型。可能的值有：string, hash, set, sortedset, list。 可以提供多个类型。如果没有指定，所有数据类型都返回。

# /usr/local/python/bin/rdb --help

Usage : rdb [ options ] / path / to / dump . rdb

Example : rdb -- command json - k "user.*" / var / redis / 6379 / dump . rdb

Options :

- h , -- help show this help message and exit

- c FILE , -- command = FILE

要执行的命令 json 或 diff

- f FILE , -- file = FILE 输出文件名

- n DBS , -- db = DBS 数据库 ID。可以提供多个数据库。如果没有指定，包含所有数据库。

- k KEYS , -- key = KEYS 导出键。可以是正则表达式。

- t TYPES , -- type = TYPES

数据类型。可能的值有： string , hash , set , sortedset , list。可以提供多个类型。如果没有指定，所有数据类型都返回。

3.1 解析dump文件并以JSON格式标准输出

# /usr/local/python/bin/rdb --command json /data/redis_data/6379/dump.rdb

1	# /usr/local/python/bin/rdb --command json /data/redis_data/6379/dump.rdb

3.2 只解析符合正则的keys

# /usr/local/python/bin/rdb --command json --key "sences_2.*" /data/redis_data/6379/dump.rdb

1	# /usr/local/python/bin/rdb --command json --key "sences_2.*" /data/redis_data/6379/dump.rdb

3.3 只解析以“a”为开头的hash且位于数据库ID为2的

# /usr/local/python/bin/rdb --command json --db 2 --type hash --key "a.*" /data/redis_data/6379/dump.rdb

1	# /usr/local/python/bin/rdb --command json --db 2 --type hash --key "a.*" /data/redis_data/6379/dump.rdb

四. 生成内存报告

生成CSV格式的内存报告。包含的列有：数据库ID，数据类型，key，内存使用量(byte)，编码。内存使用量包含key、value和其他值。
注意：内存使用量是近似的。在一般情况下，略低于实际值。
可以根据key或数据库ID或数据类型对报告的内容进行过滤。

内存报告有助于检测是否是应用程序逻辑导致的内存泄露，也有助于优化reids内存使用情况。

# /usr/local/python/bin/rdb -c memory /data/redis_data/6379/dump.rdb > redis_memory_report.csv
内容如下所示：
database,type,key,size_in_bytes,encoding,num_elements,len_largest_element
0,string,"ot_0_3b0703c01015ce05f76ef4b977dc020e820d0692",351,string,184,184
0,hash,"sences_98558",1703,hashtable,10,132
0,hash,"sences_170989",1698,hashtable,10,138
0,hash,"sences_34233",1673,hashtable,10,115
0,hash,"sence_messages2_favor_32783",358,ziplist,7,51

# /usr/local/python/bin/rdb -c memory /data/redis_data/6379/dump.rdb > redis_memory_report.csv

内容如下所示：

database , type , key , size_in_bytes , encoding , num_elements , len_largest _element

0 , string , "ot_0_3b0703c01015ce05f76ef4b977dc020e820d0692" , 351 , string , 184 , 184

0 , hash , "sences_98558" , 1703 , hashtable , 10 , 132

0 , hash , "sences_170989" , 1698 , hashtable , 10 , 138

0 , hash , "sences_34233" , 1673 , hashtable , 10 , 115

0 , hash , "sence_messages2_favor_32783" , 358 , ziplist , 7 , 51

五. 单个key所使用的内存量

有时候，需要查询某个key所使用的内存。如果全部导出来在查找将是很愚蠢且耗时的。对于这种情景，可以使用redis-memory-for-key命令。
如果出现下面信息，需要安装redis模块。redis-memory-for-key依赖redis-py包。

Traceback (most recent call last):
 File "/usr/local/python/bin/redis-memory-for-key", line 8, in <module>
 load_entry_point('rdbtools==0.1.5', 'console_scripts', 'redis-memory-for-key')()
 from redis import StrictRedis
ImportError: No module named redis

Traceback ( most recent call last ) :

File "/usr/local/python/bin/redis-memory-for-key" , line 8 , in < module >

load_entry_point ( 'rdbtools==0.1.5' , 'console_scripts' , 'redis-memory-for-key' ) ( )

from redis import StrictRedis

ImportError : No module named redis

# /usr/local/python/bin/pip install redis
或
# /usr/local/python/bin/easy_install redis

# /usr/local/python/bin/pip install redis

或

# /usr/local/python/bin/easy_install redis

# /usr/local/python/bin/redis-memory-for-key --help
Usage: redis-memory-for-key [options] redis-key
Examples :
redis-memory-for-key user:13423
redis-memory-for-key -h localhost -p 6379 user:13423
Options:
  -h, --help            show this help message and exit
  -s HOST, --server=HOST
                        Redis Server hostname. Defaults to 127.0.0.1
  -p PORT, --port=PORT  Redis Server port. Defaults to 6379
  -a PASSWORD, --password=PASSWORD
                        Password to use when connecting to the server
  -d DB, --db=DB        Database number, defaults to 0

# /usr/local/python/bin/redis-memory-for-key --help

Usage : redis - memory - for - key [ options ] redis - key

Examples :

redis - memory - for - key user : 13423

redis - memory - for - key - h localhost - p 6379 user : 13423

Options :

- h , -- help show this help message and exit

- s HOST , -- server = HOST

Redis Server hostname . Defaults to 127.0.0.1

- p PORT , -- port = PORT Redis Server port . Defaults to 6379

- a PASSWORD , -- password = PASSWORD

Password to use when connecting to the server

- d DB , -- db = DB Database number , defaults to 0

实例如下：

# /usr/local/python/bin/redis-memory-for-key -s 10.1.242.124   sence_167989
Key                             "sence_167989"
Bytes                           2712.0
Type                            hash
Encoding                        hashtable
Number of Elements              15
Length of Largest Element       222

# /usr/local/python/bin/redis-memory-for-key -s 10.1.242.124 sence_167989

Key "sence_167989"

Bytes 2712.0

Type hash

Encoding hashtable

Number of Elements 15

Length of Largest Element 222

六. 比较RDB文件

使用--command diff选项，并通过管道来进行排序。

# /usr/local/python/bin/rdb --command diff /data/redis_data/6379/dump.rdb | sort > dump1.txt
# /usr/local/python/bin/rdb --command diff /data/redis_data/6379/dump.rdb | sort > dump2.txt
# diff dump1.txt dump2.txt

# /usr/local/python/bin/rdb --command diff /data/redis_data/6379/dump.rdb | sort > dump1.txt

# /usr/local/python/bin/rdb --command diff /data/redis_data/6379/dump.rdb | sort > dump2.txt

# diff dump1.txt dump2.txt

使用kdiff3工具来进行比较，kdiff3是图形化的工具，比较直观。kdiff3工具比较两个或三个输入文件或目录。
安装kdiff3

# rpm -ivh http://dl.fedoraproject.org/pub/epel/5/x86_64/epel-release-5-4.noarch.rpm
# yum install kdiff3
# kdiff3 dump1.txt dump2.txt

# rpm -ivh http://dl.fedoraproject.org/pub/epel/5/x86_64/epel-release-5-4.noarch.rpm

# yum install kdiff3

# kdiff3 dump1.txt dump2.txt

七. 使用解析器

import sys
from rdbtools import RdbParser, RdbCallback

class MyCallback(RdbCallback) :
    ''' Simple example to show how callback works. 
        See RdbCallback for all available callback methods.
        See JsonCallback for a concrete example
    ''' 
    def set(self, key, value, expiry):
        print('%s = %s' % (str(key), str(value)))

    def hset(self, key, field, value):
        print('%s.%s = %s' % (str(key), str(field), str(value)))

    def sadd(self, key, member):
        print('%s has {%s}' % (str(key), str(member)))

    def rpush(self, key, value) :
        print('%s has [%s]' % (str(key), str(value)))

    def zadd(self, key, score, member):
        print('%s has {%s : %s}' % (str(key), str(member), str(score)))

callback = MyCallback()
parser = RdbParser(callback)
parser.parse('/var/redis/6379/dump.rdb')

import sys

from rdbtools import RdbParser , RdbCallback

class MyCallback ( RdbCallback ) :

'' ' Simple example to show how callback works.

See RdbCallback for all available callback methods.

See JsonCallback for a concrete example

' ''

def set ( self , key , value , expiry ) :

print ( '%s = %s' % ( str ( key ) , str ( value ) ) )

def hset ( self , key , field , value ) :

print ( '%s.%s = %s' % ( str ( key ) , str ( field ) , str ( value ) ) )

def sadd ( self , key , member ) :

print ( '%s has {%s}' % ( str ( key ) , str ( member ) ) )

def rpush ( self , key , value ) :

print ( '%s has [%s]' % ( str ( key ) , str ( value ) ) )

def zadd ( self , key , score , member ) :

print ( '%s has {%s : %s}' % ( str ( key ) , str ( member ) , str ( score ) ) )

callback = MyCallback ( )

parser = RdbParser ( callback )

parser . parse ( '/var/redis/6379/dump.rdb' )

八. 其他资源

1. FAQ：https://github.com/sripathikrishnan/redis-rdb-tools/wiki/FAQs
2. redis dump文件规范： https://github.com/sripathikrishnan/redis-rdb-tools/wiki/Redis-RDB-Dump-File-Format
3. redis RDB历史版本： https://github.com/sripathikrishnan/redis-rdb-tools/blob/master/docs/RDB_Version_History.textile
4. redis-rdb-tools：https://github.com/sripathikrishnan/redis-rdb-tools

九. 其他工具Redis-audit

Redis-audit 是一个用ruby实现的脚本，通过它，我们可以知道每一类 key 对内存的使用量。它可以提供的数据有：某一类 key 值的访问频率如何，有多少值设置了过期时间，某一类 key 值使用内存的大小，这很方便让我们能排查哪些 key 不常用或者压根不用。
项目地址：https://github.com/snmaynard/redis-audit

如需转载请注明出处：http://www.ttlsa.com/html/2653.html

转载请注明：成长的对话 » redis-rdb-tools来解析分析reids dump文件及内存使用量

与本文相关的文章