Docker存储驱动devicemapper介绍和配置

缥帙各舒散，前后互相逾。这篇文章主要讲述Docker存储驱动devicemapper介绍和配置相关的知识，希望能为你提供帮助。
一.devicemapper介绍

Device Mapper是Linux系统中基于内核的高级卷管理技术框架。Docker的devicemapper存储驱动就是基于该框架的精简置备和快照功能来实现镜像和容器的管理。注:Device Mapper是Linux的一种技术框架，而devicemapper是Docker Engine基于Device Mapper提供的一种存储驱动。早期的Docker运行在Ubuntu和Debian Linux上并使用AUFS作为后端存储。Docker流行之后，越来越多的的公司希望在Red Hat Enterprise Linux这类企业级的操作系统上面运行Docker，但可惜的是RHEL的内核并不支持AUFS。这个时候红帽公司出手了，决定和Docker公司合作去开发一种基于Device Mapper技术的后端存储，也就是现在的devicemapper。devicemapper驱动将每一个Docker镜像和容器存储在它自身的具有精简置备(thin-provisioned)、写时拷贝(copy-on-write)和快照功能(snapshotting)的虚拟设备上。由于Device Mapper技术是在块(block)层面而非文件层面，所以Docker Engine的devicemapper存储驱动使用的是块设备来存储数据而非文件系统。

二.devicemapper的模式

devicemapper是RHEL下Docker Engine的默认存储驱动，它有两种配置模式:loop-lvm和direct-lvm。loop-lvm是默认的模式，它使用OS层面离散的文件来构建精简池(thin pool)。该模式主要是设计出来让Docker能够简单的被”开箱即用(out-of-the-box)”而无需额外的配置。但如果是在生产环境的部署Docker，官方明文不推荐使用该模式。我们使用docker info命令可以看到以下警告: WARNING: Usage of loopback devices is strongly discouraged for production use. Either use `–storage-opt dm.thinpooldev` or use `–storage-opt dm.no_warn_on_loop_devices=true` to suppress this warning. direct-lvm是Docker推荐的生产环境的推荐模式，他使用块设备来构建精简池来存放镜像和容器的数据。前段时间有篇很不错的微信文章是关于老司机填devicemapper坑的血泪史，仔细研读之后发现老司机使用的是loop-lvm模式，那个坑有可能由此引起，最终老司机使用overlayfs的存储驱动解决了问题。注:Linux内核在3.18以上才能支持overlayfs，但RHEL 7.2的内核版本为3.10，所以原生并不支持。但是的确有人在RHEL7.2上成功应用了overlayfs驱动，个人猜测可能是手动在内核里面加载了overlay的模块。

三.配置direct-lvm模式

1.停止Docker并备份

如果Docker服务已在运行且有需要保留的镜像和容器，停服务前把相关数据给备份。个人也强烈建议如果是在生产环境使用Docker的话，拿到host的第一时间就将direct-lvm模式给配置了。(当然也可以选择其他的storage driver)

2.查看当前devicemapper模式

[[email protected] ~]# docker info Containers: 0 Running: 0 Paused: 0 Stopped: 0 Images: 0 Server Version: 17.12.0-ce Storage Driver: devicemapper Pool Name: docker-8:3-1073035-pool Pool Blocksize: 65.54kB Base Device Size: 10.74GB Backing Filesystem: xfs Udev Sync Supported: true Data file: /dev/loop0 Metadata file: /dev/loop1 Data loop file: /var/lib/docker/devicemapper/devicemapper/data Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata Data Space Used: 11.8MB Data Space Total: 107.4GB Data Space Available: 51.36GB Metadata Space Used: 581.6kB Metadata Space Total: 2.147GB Metadata Space Available: 2.147GB Thin Pool Minimum Free Space: 10.74GB Deferred Removal Enabled: true Deferred Deletion Enabled: true Deferred Deleted Device Count: 0 Library Version: 1.02.140-RHEL7 (2017-05-03) Logging Driver: json-file Cgroup Driver: cgroupfs Plugins: Volume: local Network: bridge host macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog Swarm: inactive Runtimes: runc Default Runtime: runc Init Binary: docker-init containerd version: 89623f28b87a6004d4b785663257362d1658a729 runc version: b2567b37d7b75eb4cf325b77297b140ea686ce8f init version: 949e6fa Security Options: seccomp Profile: default Kernel Version: 3.10.0-327.el7.x86_64 Operating System: CentOS Linux 7 (Core) OSType: linux Architecture: x86_64 CPUs: 2 Total Memory: 1.954GiB Name: docker ID: IMMY:YLYX:LF5E:GZID:ACCP:4V43:2IPT:MCSD:DINH:MKFJ:DSDV:TWF4 Docker Root Dir: /var/lib/docker Debug Mode (client): false Debug Mode (server): false Registry: https://index.docker.io/v1/ Labels: Experimental: false Insecure Registries: 127.0.0.0/8 Live Restore Enabled: falseWARNING: devicemapper: usage of loopback devices is strongly discouraged for production use. Use `--storage-opt dm.thinpooldev` to specify a custom block storage device. [[email protected] ~]#

基于docker info查询的结果可以看到当前模式为loop-lvm。

3.停止docker服务

[[email protected] ~]# systemctl stop docker

四.分配裸设备

本例以添加硬盘到docker宿主机，推荐使用外部共享存储的设备但不局限于此种方式，可根据自己的环境决定。

添加200GB的硬盘
创建一个Volume Group
将Volume Group挂给docker宿主机

创建VG
1.查看设备

[[email protected] ~]# fdisk -l /dev/sdbDisk /dev/sdb: 214.7 GB, 214748364800 bytes, 419430400 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes[[email protected] ~]#

2.创建PV

[[email protected] ~]# pvcreate /dev/sdb Physical volume " /dev/sdb" successfully created. [[email protected] ~]#

3.创建VG

[[email protected] ~]# vgcreate docker /dev/sdb Volume group " docker" successfully created [[email protected] ~]#

4.查看VG信息

[[email protected] ~]# vgs VG #PV #LV #SN Attr VSize VFree docker 1 0 0 wz--n- < 200.00g < 200.00g [[email protected] ~]# vgdisplay docker --- Volume group --- VG Name docker System ID Format lvm2 Metadata Areas 1 Metadata Sequence No 1 VG Access read/write VG Status resizable MAX LV 0 Cur LV 0 Open LV 0 Max PV 0 Cur PV 1 Act PV 1 VG Size < 200.00 GiB PE Size 4.00 MiB Total PE 51199 Alloc PE / Size 0 / 0 Free PE / Size 51199 / < 200.00 GiB VG UUID m6or9g-k3Ff-s5eX-w9yz-MjyN-oaA5-l2Vwsw [[email protected] ~]#

创建thinpool

创建pool

[[email protected] ~]# lvcreate --wipesignatures y -n thinpool docker -l 95%VG Logical volume " thinpool" created. [[email protected] ~]# lvcreate --wipesignatures y -n thinpoolmeta docker -l 1%VG Logical volume " thinpoolmeta" created. [[email protected] ~]#

数据LV大小为VG的95%,元数据LV大小为VG的1%,剩余的空间用来自动扩展。

2.将pool转换为thinpool

[[email protected] ~]# lvcreate --wipesignatures y -n thinpoolmeta docker -l 1%VG Logical volume " thinpoolmeta" created. [[email protected] ~]# lvconvert -y --zero n -c 512K --thinpool docker/thinpool --poolmetadata docker/thinpoolmeta Thin pool volume with chunk size 512.00 KiB can address at most 126.50 TiB of data. WARNING: Converting logical volume docker/thinpool and docker/thinpoolmeta to thin pool' s data and metadata volumes with metadata wiping. THIS WILL DESTROY CONTENT OF LOGICAL VOLUME (filesystem etc.) Converted docker/thinpool_tdata to thin pool. [[email protected] ~]#

配置thinpool

配置池的自动扩展

[[email protected] ~]# cat /etc/lvm/profile/docker-thinpool.profile activation { thin_pool_autoextend_threshold=80 thin_pool_autoextend_percent=20 } [[email protected] ~]#

2.应用配置变更

[[email protected] ~]# lvchange --metadataprofile docker-thinpool docker/thinpool Logical volume docker/thinpool changed. [[email protected] ~]#

3.状态监控检查

[[email protected] ~]# lvs -o+seg_monitor LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert Monitor thinpool docker twi-a-t--- < 190.00g 0.00 0.01 monitored [[email protected] ~]#

配置Docker
1.修改服务配置文件

[[email protected] ~]# vim /usr/lib/systemd/system/docker.service --storage-driver=devicemapper --storage-opt=dm.thinpooldev=/dev/mapper/docker-thinpool --storage-opt dm.use_deferred_removal=true

ExecStart后加入storage相关配置参数,如果配置了$OPTIONS也可以在对应的EnvironmentFile中加入。

2.清除graphdriver

[[email protected] ~]# rm -rf /var/lib/docker/*

之前已提醒数据备份，因为在这里清除graphdriver会将image,Container和volume所有数据都删除。如果不删除，则会遇到以下的错误导致docker服务起不来的。

Error starting daemon: error initializing graphdriver: devmapper: Base Device UUID and Filesystem verification failed: devicemapper: Error running deviceCreate (ActivateDevice) dm_task_run failed

启动docker服务

[[email protected] ~]# systemctl daemon-reload [[email protected] ~]# systemctl start docker

检查devicemapper配置

[[email protected] ~]# docker info Containers: 0 Running: 0 Paused: 0 Stopped: 0 Images: 0 Server Version: 17.12.0-ce Storage Driver: devicemapper Pool Name: docker-thinpool Pool Blocksize: 524.3kB Base Device Size: 10.74GB Backing Filesystem: xfs Udev Sync Supported: true Data Space Used: 20.45MB Data Space Total: 204GB Data Space Available: 204GB Metadata Space Used: 266.2kB Metadata Space Total: 2.143GB Metadata Space Available: 2.143GB Thin Pool Minimum Free Space: 20.4GB Deferred Removal Enabled: true Deferred Deletion Enabled: true Deferred Deleted Device Count: 0 Library Version: 1.02.140-RHEL7 (2017-05-03) Logging Driver: json-file Cgroup Driver: cgroupfs Plugins: Volume: local Network: bridge host macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog Swarm: inactive Runtimes: runc Default Runtime: runc Init Binary: docker-init containerd version: 89623f28b87a6004d4b785663257362d1658a729 runc version: b2567b37d7b75eb4cf325b77297b140ea686ce8f init version: 949e6fa Security Options: seccomp Profile: default Kernel Version: 3.10.0-327.el7.x86_64 Operating System: CentOS Linux 7 (Core) OSType: linux Architecture: x86_64 CPUs: 2 Total Memory: 1.954GiB Name: docker ID: IMMY:YLYX:LF5E:GZID:ACCP:4V43:2IPT:MCSD:DINH:MKFJ:DSDV:TWF4 Docker Root Dir: /var/lib/docker Debug Mode (client): false Debug Mode (server): false Registry: https://index.docker.io/v1/ Labels: Experimental: false Insecure Registries: 127.0.0.0/8 Live Restore Enabled: false[[email protected] ~]#

基于docker info查询的结果可以看到当前模式为direct-lvm。

测试
pull一个镜像看是否数据会写到thinpool里；

[[email protected] ~]# lvs LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert thinpool docker twi-a-t--- < 190.00g 0.01 0.01 [[email protected] ~]# docker pull centos Using default tag: latest latest: Pulling from library/centos af4b0a2388c6: Pull complete Digest: sha256:2671f7a3eea36ce43609e9fe7435ade83094291055f1c96d9d1d1d7c0b986a5d Status: Downloaded newer image for centos:latest [[email protected] ~]# lvs LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert thinpool docker twi-a-t--- < 190.00g 0.13 0.01 [[email protected] ~]#

【Docker存储驱动devicemapper介绍和配置】可以看到Data%在pull一个centos镜像后使用率由0.01变为0.13，说明direct-lvm配置成功且正常工作。