Redis单机实例数据迁移完整步骤
一、源码部署Redis单机实例
1、下载指定版本Redis源码，本文Redis 4.0.12（模拟源Redis数据实例）。
$ wget http://download.redis.io/releases/redis-4.0.12.tar.gz
$ tar xzf redis-4.0.12.tar.gz
$ cd redis-4.0.12
$ apt install gcc && make && make install2、下载指定版本Redis源码，本文Redis 5.0.12（模拟新Redis数据实例）。
$ wget http://download.redis.io/releases/redis-5.0.12.tar.gz
$ tar xzf redis-5.0.12.tar.gz
$ cd redis-5.0.12
$ apt install gcc make -y && make && make install
二、分别配置相关Redis实例
1、导入实验数据库，这里任意导入appendonly.aof 或 dump.rdb，其中之一即可。
## 导入appendonly.aof备份文件为例
$ redis-cli --pipe < appendonly.aof
18279:M 24 Feb 11:16:31.264 * 1 changes in 900 seconds. Saving...
18279:M 24 Feb 11:16:31.265 * Background saving started by pid 27879
27879:C 24 Feb 11:16:31.345 * DB saved on disk
27879:C 24 Feb 11:16:31.352 * RDB: 0 MB of memory used by copy-on-write
18279:M 24 Feb 11:16:31.373 * Background saving terminated with success
All data transferred. Waiting for the last reply...
Last reply received from server.
errors: 0, replies: 2678225Redis 清空命令：redis-cli -h 127.0.0.1 -a password flushall
Redis开/关aof功能：redis-cli -h 127.0.0.1 -a password config set appendonly no
## 导入dump.rdb备份文件为例redis-dump 麻烦就麻烦在需要进行安装，如果我的 Redis 已经有备份机制，比如有 rdb 文件，那么我们直接迁移 rdb 文件就可以达到同样的目的。
首先，我们可以先关闭源 Redis 实例的 aof 功能。如果不关闭 aof，Redis 默认用 aof 文件来恢复数据。
# 源实例关闭 aof 功能
redis-cli -h 源RedisIp -a password config set appendonly no# 固化数据到 RDB 文件
save
# 查看dump.rdb文件路径
config get dir接下来，我们需要杀死 redis 进程。杀掉当前 redis 的进程，否则下一步的复制 rdb 文件，rdb 处于打开的状态，复制的文件，会占用同样的句柄。
killall redis-server
# 或者
pkill -9 redis
# 或者手段关闭 Redis 服务最后，复制替换Redis备份文件dump.rdb，重启目标 Redis 实例，数据就迁移完成了。
三、重点，Redis单机如何迁移数据至新的Redis实例。
1、方案一、shell脚本实现源实例指定“db0”迁移至目标Redis实例“db1”
#!/bin/bash
redis-cli -h 10.100.202.224 -p 6379 -n 0 keys "*" | while read key
do
redis-cli -h 10.100.202.224 -p 6379 -n 0 --raw dump $key | perl -pe 'chomp if eof' | redis-cli -h 10.100.202.161 -p 6379 -n 1 -x restore $key 0
echo "migrate key $key"
done
## Redis热迁移10.100.202.224至10.100.202.161，并指定db库。2、方案二、基于阿里云Redis开源工具实现
首先，下载Redis-Shake，Redis同步工具源码
$ wget https://github.com/alibaba/RedisShake/releases/download/release-v2.1.1-20210903/release-v2.1.1-20210903.tar.gz
$ tar xf release-v2.1.1-20210903.tar.gz -C /usr/local/
$ cd /usr/local/release-v2.1.1-20210903/然后，配置Redis-Shake工具
# This file is the configuration of redis-shake.
# If you have any problem, please visit: https://github.com/alibaba/RedisShake/wiki/FAQ
# 有疑问请先查阅：https://github.com/alibaba/RedisShake/wiki/FAQ
# current configuration version, do not modify.
# 当前配置文件的版本号，请不要修改该值。
conf.version = 1
# id
id = redis-shake
# The log file name, if left blank, it will be printed to stdout,
# otherwise it will be printed to the specified file.
# 日志文件名，留空则会打印到 stdout，否则打印到指定文件。
# for example:
# log.file =
# log.file = /var/log/redis-shake.log
log.file = /var/log/redis-shake.log
# log level: "none", "error", "warn", "info", "debug".
# default is "info".
# 日志等级，可选：none error warn info debug
# 默认为：info
log.level = info
# 进程文件存储目录，留空则会输出到当前目录,
# 注意这个是目录，真正生成的 pid 是 {pid_path}/{id}.pid
# 例如：
# pid_path = ./
# pid_path = /var/run/
pid_path = /var/run/
# pprof port.
system_profile = 9310
# restful port, set -1 means disable, in `restore` mode RedisShake will exit once finish restoring RDB only if this value
# is -1, otherwise, it'll wait forever.
# restful port, 查看 metric 端口, -1 表示不启用. 如果是`restore`模式，只有设置为-1才会在完成RDB恢复后退出，否则会一直block。
# http://127.0.0.1:9320/conf 查看 redis-shake 使用的配置
# http://127.0.0.1:9320/metric 查看 redis-shake 的同步情况
http_profile = 9320
# parallel routines number used in RDB file syncing. default is 64.
# 启动多少个并发线程同步一个RDB文件。
parallel = 32
# source redis configuration.
# used in `dump`, `sync` and `rump`.
# source redis type, e.g. "standalone" (default), "sentinel" or "cluster".
# 1. "standalone": standalone db mode.
# 2. "sentinel": the redis address is read from sentinel.
# 3. "cluster": the source redis has several db.
# 4. "proxy": the proxy address, currently, only used in "rump" mode.
# used in `dump`, `sync` and `rump`.
# 源端 Redis 的类型，可选：standalone sentinel cluster proxy
# 注意：proxy 只用于 rump 模式。
source.type = standalone
# ip:port
# the source address can be the following:
# 1. single db address. for "standalone" type.
# 2. ${sentinel_master_name}:${master or slave}@sentinel single/cluster address, e.g., mymaster:master@127.0.0.1:26379;127.0.0.1:26380, or @127.0.0.1:26379;127.0.0.1:26380. for "sentinel" type.
# 3. cluster that has several db nodes split by semicolon(;). for "cluster" type. e.g., 10.1.1.1:20331;10.1.1.2:20441.
# 4. proxy address(used in "rump" mode only). for "proxy" type.
# 源redis地址。对于sentinel或者开源cluster模式，输入格式为"master名字:拉取角色为master或者slave@sentinel的地址"，别的cluster
# 架构，比如codis, twemproxy, aliyun proxy等需要配置所有master或者slave的db地址。
# 源端 redis 的地址
# 1. standalone 模式配置 ip:port, 例如: 10.1.1.1:20331
# 2. cluster 模式需要配置所有 nodes 的 ip:port, 例如: source.address = 10.1.1.1:20331;10.1.1.2:20441
# source.address = 127.0.0.1:20441
source.address = 10.100.202.224:6379
# source password, left blank means no password.
# 源端密码，留空表示无密码。
# source.password_raw =
source.password_raw = qwqw1212
# auth type, don't modify it
source.auth_type = auth
# tls enable, true or false. Currently, only support standalone.
# open source redis does NOT support tls so far, but some cloud versions do.
source.tls_enable = false
# input RDB file.
# used in `decode` and `restore`.
# if the input is list split by semicolon(;), redis-shake will restore the list one by one.
# 如果是decode或者restore，这个参数表示读取的rdb文件。支持输入列表，例如：rdb.0;rdb.1;rdb.2
# redis-shake将会挨个进行恢复。
source.rdb.input =
# the concurrence of RDB syncing, default is len(source.address) or len(source.rdb.input).
# used in `dump`, `sync` and `restore`. 0 means default.
# This is useless when source.type isn't cluster or only input is only one RDB.
# 拉取的并发度，如果是`dump`或者`sync`，默认是source.address中db的个数，`restore`模式默认len(source.rdb.input)。
# 假如db节点/输入的rdb有5个，但rdb.parallel=3，那么一次只会
# 并发拉取3个db的全量数据，直到某个db的rdb拉取完毕并进入增量，才会拉取第4个db节点的rdb，
# 以此类推，最后会有len(source.address)或者len(rdb.input)个增量线程同时存在。
source.rdb.parallel = 0
# for special cloud vendor: ucloud
# used in `decode` and `restore`.
# ucloud集群版的rdb文件添加了slot前缀，进行特判剥离: ucloud_cluster。
source.rdb.special_cloud =
# target redis configuration. used in `restore`, `sync` and `rump`.
# the type of target redis can be "standalone", "proxy" or "cluster".
# 1. "standalone": standalone db mode.
# 2. "sentinel": the redis address is read from sentinel.
# 3. "cluster": open source cluster (not supported currently).
# 4. "proxy": proxy layer ahead redis. Data will be inserted in a round-robin way if more than 1 proxy given.
# 目的redis的类型，支持standalone，sentinel，cluster和proxy四种模式。
target.type = standalone
# ip:port
# the target address can be the following:
# 1. single db address. for "standalone" type.
# 2. ${sentinel_master_name}:${master or slave}@sentinel single/cluster address, e.g., mymaster:master@127.0.0.1:26379;127.0.0.1:26380, or @127.0.0.1:26379;127.0.0.1:26380. for "sentinel" type.
# 3. cluster that has several db nodes split by semicolon(;). for "cluster" type.
# 4. proxy address. for "proxy" type.
# target.address = 127.0.0.1:6379
target.address = 10.100.202.161:6379
# target password, left blank means no password.
# 目的端密码，留空表示无密码。
target.password_raw = qwqw1212
# auth type, don't modify it
target.auth_type = auth
# all the data will be written into this db. < 0 means disable.
target.db = -1
# Format: 0-5;1-3 ,Indicates that the data of the source db0 is written to the target db5, and
# the data of the source db1 is all written to the target db3.
# Note: When target.db is specified, target.dbmap will not take effect.
# 例如 0-5;1-3 表示源端 db0 的数据会被写入目的端 db5, 源端 db1 的数据会被写入目的端 db3
# 当 target.db 开启的时候 target.dbmap 不会生效.
target.dbmap = 2-0;8-1
# tls enable, true or false. Currently, only support standalone.
# open source redis does NOT support tls so far, but some cloud versions do.
target.tls_enable = false
# output RDB file prefix.
# used in `decode` and `dump`.
# 如果是decode或者dump，这个参数表示输出的rdb前缀，比如输入有3个db，那么dump分别是:
# ${output_rdb}.0, ${output_rdb}.1, ${output_rdb}.2
target.rdb.output = local_dump
# some redis proxy like twemproxy doesn't support to fetch version, so please set it here.
# e.g., target.version = 4.0
target.version =
# use for expire key, set the time gap when source and target timestamp are not the same.
# 用于处理过期的键值，当迁移两端不一致的时候，目的端需要加上这个值
fake_time =
# how to solve when destination restore has the same key.
# rewrite: overwrite.
# none: panic directly.
# ignore: skip this key. not used in rump mode.
# used in `restore`, `sync` and `rump`.
# 当源目的有重复 key 时是否进行覆写, 可选值:
# 1. rewrite: 源端覆盖目的端
# 2. none: 一旦发生进程直接退出
# 3. ignore: 保留目的端key，忽略源端的同步 key. 该值在 rump 模式下不会生效.
key_exists = none
# filter db, key, slot, lua.
# filter db.
# used in `restore`, `sync` and `rump`.
# e.g., "0;5;10" means match db0, db5 and db10.
# at most one of `filter.db.whitelist` and `filter.db.blacklist` parameters can be given.
# if the filter.db.whitelist is not empty, the given db list will be passed while others filtered.
# if the filter.db.blacklist is not empty, the given db list will be filtered while others passed.
# all dbs will be passed if no condition given.
# 指定的db被通过，比如0;5;10将会使db0, db5, db10通过, 其他的被过滤
filter.db.whitelist =
# 指定的db被过滤，比如0;5;10将会使db0, db5, db10过滤，其他的被通过
filter.db.blacklist = 0;1;3;4;5;6;7;9;10;11;12;13;14;15
# filter key with prefix string. multiple keys are separated by ';'.
# e.g., "abc;bzz" match let "abc", "abc1", "abcxxx", "bzz" and "bzzwww".
# used in `restore`, `sync` and `rump`.
# at most one of `filter.key.whitelist` and `filter.key.blacklist` parameters can be given.
# if the filter.key.whitelist is not empty, the given keys will be passed while others filtered.
# if the filter.key.blacklist is not empty, the given keys will be filtered while others passed.
# all the namespace will be passed if no condition given.
# 支持按前缀过滤key，只让指定前缀的key通过，分号分隔。比如指定abc，将会通过abc, abc1, abcxxx
filter.key.whitelist =
# 支持按前缀过滤key，不让指定前缀的key通过，分号分隔。比如指定abc，将会阻塞abc, abc1, abcxxx
filter.key.blacklist =
# filter given slot, multiple slots are separated by ';'.
# e.g., 1;2;3
# used in `sync`.
# 指定过滤slot，只让指定的slot通过
filter.slot =
# filter lua script. true means not pass. However, in redis 5.0, the lua
# converts to transaction(multi+{commands}+exec) which will be passed.
# 控制不让lua脚本通过，true表示不通过
filter.lua = false
# big key threshold, the default is 500 * 1024 * 1024 bytes. If the value is bigger than
# this given value, all the field will be spilt and write into the target in order. If
# the target Redis type is Codis, this should be set to 1, please checkout FAQ to find
# the reason.
# 正常key如果不大，那么都是直接调用restore写入到目的端，如果key对应的value字节超过了给定
# 的值，那么会分批依次一个一个写入。如果目的端是Codis，这个需要置为1，具体原因请查看FAQ。
# 如果目的端大版本小于源端，也建议设置为1。
big_key_threshold = 524288000
# enable metric
# used in `sync`.
# 是否启用metric
# metric = true
metric = false
# print in log
# 是否将metric打印到log中
metric.print_log = false
# sender information.
# sender flush buffer size of byte.
# used in `sync`.
# 发送缓存的字节长度，超过这个阈值将会强行刷缓存发送
sender.size = 104857600
# sender flush buffer size of oplog number.
# used in `sync`. flush sender buffer when bigger than this threshold.
# 发送缓存的报文个数，超过这个阈值将会强行刷缓存发送，对于目的端是cluster的情况，这个值
# 的调大将会占用部分内存。
sender.count = 4095
# delay channel size. once one oplog is sent to target redis, the oplog id and timestamp will also
# stored in this delay queue. this timestamp will be used to calculate the time delay when receiving
# ack from target redis.
# used in `sync`.
# 用于metric统计时延的队列
sender.delay_channel_size = 65535
# enable keep_alive option in TCP when connecting redis.
# the unit is second.
# 0 means disable.
# TCP keep-alive保活参数，单位秒，0表示不启用。
keep_alive = 0
# used in `rump`.
# number of keys captured each time. default is 100.
# 每次scan的个数，不配置则默认100.
scan.key_number = 50
# used in `rump`.
# we support some special redis types that don't use default `scan` command like alibaba cloud and tencent cloud.
# 有些版本具有特殊的格式，与普通的scan命令有所不同，我们进行了特殊的适配。目前支持腾讯云的集群版"tencent_cluster"
# 和阿里云的集群版"aliyun_cluster"，注释主从版不需要配置，只针对集群版。
scan.special_cloud =
# used in `rump`.
# we support to fetching data from given file which marks the key list.
# 有些云版本，既不支持sync/psync，也不支持scan，我们支持从文件中进行读取所有key列表并进行抓取：一行一个key。
scan.key_file =
# limit the rate of transmission. Only used in `rump` currently.
# e.g., qps = 1000 means pass 1000 keys per second. default is 500,000(0 means default)
qps = 200000
# enable resume from break point, please visit xxx to see more details.
# 断点续传开关
resume_from_break_point = false
# ----------------splitter----------------
# below variables are useless for current open source version so don't set.
# replace hash tag.
# used in `sync`.
replace_hash_tag = false
最后，启动Redis-Shake同步工具
## xxx为sync, restore, dump, decode, rump其中之一，全量+增量同步请选择sync。
## mac下请使用redis-shake.darwin，windows请用redis-shake.windows.
$ ./redis-shake.linux -conf=redis-shake.conf -type=sync
常用参数说明
| 配置项 | 说明 |
|---|---|
| target.db | 设置待迁移的数据在目的Redis中的逻辑数据库名。例如，要将所有数据迁移到目的Redis中的DB10，则需将此参数的值设置为10。当该值设置为-1时，逻辑数据库名在源Redis和目的Redis中的名称相同，即源Redis中的DB0将被迁移至目的Redis中的DB0，DB1将被迁移至DB1，以此类推。 |
| filter.db.whitelist | 指定的db被通过，比如0;5;10将会使db0, db5, db10通过, 其他的被过滤 |
| filter.db.blacklist | 指定的db被过滤，比如0;5;10将会使db0, db5, db10过滤，其他的被通过 |
| key_exists | 当源目的有重复key，是否进行覆写。rewrite表示源端覆盖目的端。none表示一旦发生进程直接退出。ignore表示保留目的端key，忽略源端的同步key。该值在rump模式下没有用。 |
四、日志校验
同步分为三个阶段：
1、等待源端save rdb完毕，日志如下
2022/02/24 15:44:47 [INFO] DbSyncer[0] psync connect '10.100.202.224:6379' with auth type[auth] OK!
2022/02/24 15:44:47 [INFO] DbSyncer[0] psync send listening port[9320] OK!
2022/02/24 15:44:47 [INFO] DbSyncer[0] try to send 'psync' command: run-id[?], offset[-1]
2、全量同步阶段，显示百分比
2022/02/24 15:44:48 [INFO] DbSyncer[0] total = 11.18MB - 11.17MB [ 99%] entry=6889 filter=166863、增量同步，出现字样sync rdb done后，当前dbSyncer进入增量同步
2022/02/24 15:44:49 [INFO] DbSyncer[0] total = 11.18MB - 11.18MB [100%] entry=8151 filter=16686
2022/02/24 15:44:49 [INFO] DbSyncer[0] sync rdb done
2022/02/24 15:44:49 [INFO] DbSyncer[0] FlushEvent:IncrSyncStart Id:redis-shake
2022/02/24 15:44:50 [INFO] DbSyncer[0] sync: +forwardCommands=1 +filterCommands=0 +writeBytes=4
2022/02/24 15:44:51 [INFO] DbSyncer[0] sync: +forwardCommands=0 +filterCommands=0 +writeBytes=0
2022/02/24 15:44:52 [INFO] DbSyncer[0] sync: +forwardCommands=0 +filterCommands=0 +writeBytes=0
2022/02/24 15:44:53 [INFO] DbSyncer[0] sync: +forwardCommands=0 +filterCommands=0 +writeBytes=0
2022/02/24 15:44:54 [INFO] DbSyncer[0] sync: +forwardCommands=0 +filterCommands=0 +writeBytes=0
2022/02/24 15:44:55 [INFO] DbSyncer[0] sync: +forwardCommands=0 +filterCommands=0 +writeBytes=0
2022/02/24 15:44:56 [INFO] DbSyncer[0] sync: +forwardCommands=0 +filterCommands=0 +writeBytes=0
2022/02/24 15:44:57 [INFO] DbSyncer[0] sync: +forwardCommands=2 +filterCommands=0 +writeBytes=36
2022/02/24 15:44:58 [INFO] DbSyncer[0] sync: +forwardCommands=0 +filterCommands=0 +writeBytes=0
2022/02/24 15:44:59 [INFO] DbSyncer[0] sync: +forwardCommands=1 +filterCommands=0 +writeBytes=4
2022/02/24 15:45:00 [INFO] DbSyncer[0] sync: +forwardCommands=0 +filterCommands=0 +writeBytes=0
2022/02/24 15:45:01 [INFO] DbSyncer[0] sync: +forwardCommands=1 +filterCommands=0 +writeBytes=29
Redis单机实例数据迁移完整步骤
https://cn.10691.cn//archives/10063