Ubuntu 12.04配置OCFS2

By , June 14, 2013 3:23 pm

简单地作个笔记。

测试环境包含三台服务器。其中一台服务器配置iSCSI TGT山寨一个IP SAN服务,将一块本地硬盘通过iSCSI协议暴露给所有的服务器。三台服务器均通过openi-scsi发现山寨IP SAN所提供的iSCSI存储。具体的配置方法可以参见《在Ubuntu 12.04上配置iSCSI Target服务》一文。

安装OCFS2软件:

sudo apt-get install ocfs2-tools

创建配置文件/etc/ocfs2/cluster.conf,内容如下(需要根据你的配置替换掉配置文件中的IP地址和主机名)。

node:
        ip_port = 7777
        ip_address = 192.168.100.20
        number = 0
        name = node0
        cluster = ocfs2
node:
        ip_port = 7777
        ip_address = 192.168.100.19
        number = 1
        name = node1
        cluster = ocfs2
node:
        ip_port = 7777
        ip_address = 192.168.100.18
        number = 2
        name = node2
        cluster = ocfs2
cluster:
        node_count = 3
        name = ocfs2

启动相应的服务:

sudo chkconfig o2cb on
sudo chkconfig ocfs2 on
sudo /etc/init.d/o2cb restart
sudo /etc/init.d/ocfs2 restart

在iSCSI TGT服务器上将通过iSCSI协议暴露出去的硬盘上创建一个分区,并格式化为OCFS2文件系统(假设该分区所对应的设备名为/dev/sdc1):

sudo mkfs.ocfs2 /dev/sdc1

在所有的服务器上挂载对应的分区到/data目录:

sudo mkdir /data
sudo mount -t ocfs2 /dev/sdc1 /data

也可以将对应的配置写入/etc/fstab文件,以便在操作系统启动时自动挂载:

/dev/sdc1        /data        ocfs2        _netdev        0        0

下面利用iozone来针对使用NFS方案和OCFS2方案的虚拟机文件IO性能做一个初步的对比。在NFS方案中,将服务器A 本地磁盘上的一个目录通过NFS服务共享给计算节点B,虚拟机实例运行在计算节点B 上,其磁盘映像文件存放在NFS上。在OCFS2方案中,将服务器A 本地磁盘上的一个分区通过iSCSI TGT服务暴露给所有的服务器,并在此基础上配置OCFS2集群文件系统,虚拟机实例运行在计算节点B 上,其磁盘映像文件存放在OCFS2上。存储服务器和计算节点的配置都是单颗4 核处理器(Intel(R) Core(TM) i5-3330S CPU @ 2.70GHz),8 GB内存,750GB的7200转SATA硬盘,千兆网卡,运行64位Ubuntu 12.04服务器版+KVM进行虚拟化。虚拟机的配置为2 个处理器核,4 GB内存,10 GB磁盘,运行64位Ubuntu 12.04服务器版,磁盘和网卡都启用了virtio,并且设置虚拟机磁盘缓存参数为cache=”none”。

在NFS方案下得到的虚拟机文件IO测试数据:

ubuntu@ubuntu:~/iozone3_414/src/current$ sudo ./iozone -Mcew -i0 -i1 -s4g -r256k -f /io.tmp
	Iozone: Performance Test of File I/O
	        Version $Revision: 3.414 $
		Compiled for 64 bit mode.
		Build: linux 

	Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins
	             Al Slater, Scott Rhine, Mike Wisner, Ken Goss
	             Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR,
	             Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner,
	             Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Dave Boone,
	             Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root,
	             Fabrice Bacchella, Zhenghua Xue, Qin Li, Darren Sawyer,
	             Vangel Bojaxhi, Ben England.

	Run began: Fri Jun 14 14:59:24 2013


	Machine = Linux ubuntu 3.2.0-29-generic #46-Ubuntu SMP Fri Jul 27 17:03:23 UTC	Include close in write timing
	Include close in write timing
	Include fsync in write timing
	Setting no_unlink
	File size set to 4194304 KB
	Record Size 256 KB
	Command line used: ./iozone -Mcew -i0 -i1 -s4g -r256k -f /io.tmp
	Output is in Kbytes/sec
	Time Resolution = 0.000001 seconds.
	Processor cache size set to 1024 Kbytes.
	Processor cache line size set to 32 bytes.
	File stride size set to 17 * record size.
                                                            random  random    bkwd   record   stride                                   
              KB  reclen   write rewrite    read    reread    read   write    read  rewrite     read   fwrite frewrite   fread  freread
         4194304     256   36678   26711   160948   163411                                                                          

iozone test complete.

在OCFS2方案下得到的虚拟机文件IO测试数据:

ubuntu@ubuntu:~/iozone3_414/src/current$ sudo ./iozone -Mcew -i0 -i1 -s4g -r256k -f /io.tmp
	Iozone: Performance Test of File I/O
	        Version $Revision: 3.414 $
		Compiled for 64 bit mode.
		Build: linux 

	Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins
	             Al Slater, Scott Rhine, Mike Wisner, Ken Goss
	             Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR,
	             Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner,
	             Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Dave Boone,
	             Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root,
	             Fabrice Bacchella, Zhenghua Xue, Qin Li, Darren Sawyer,
	             Vangel Bojaxhi, Ben England.

	Run began: Fri Jun 14 15:30:05 2013


	Machine = Linux ubuntu 3.2.0-29-generic #46-Ubuntu SMP Fri Jul 27 17:03:23 UTC	Include close in write timing
	Include close in write timing
	Include fsync in write timing
	Setting no_unlink
	File size set to 4194304 KB
	Record Size 256 KB
	Command line used: ./iozone -Mcew -i0 -i1 -s4g -r256k -f /io.tmp
	Output is in Kbytes/sec
	Time Resolution = 0.000001 seconds.
	Processor cache size set to 1024 Kbytes.
	Processor cache line size set to 32 bytes.
	File stride size set to 17 * record size.
                                                            random  random    bkwd   record   stride                                   
              KB  reclen   write rewrite    read    reread    read   write    read  rewrite     read   fwrite frewrite   fread  freread
         4194304     256   59251   60797   140720   148803                                                                          

iozone test complete.

如下是一组关于物理机和虚拟机的对比测试数据。这组测试虽然粗糙,但还是能够定性地说明OCFS2的性能比NFS是强不止一点点的。

测试环境 write rewrite read reread
存储节点 | 本地硬盘 122684 119326 8564609 8792814
计算节点 | NFS 63184 66472 95322 6738489
计算节点 | OCFS2 75547 82115 175872 7654664
虚拟机 | NFS 36678 26711 160948 163411
虚拟机 | OCFS2 59251 60797 140720 148803

相关文档:《OCFS2 - A CLUSTER FILE SYSTEM FOR LINUX》

One Response to “Ubuntu 12.04配置OCFS2”

  1. [...] 在我的前一篇博客文章《Ubuntu 12.04配置OCFS2》中对在NFS和OCFS2上分别运行虚拟机的文件IO性能做了一些类似的测试,在这里将NFS、GFS2和OCFS2的测试结果综合如下: [...]

Leave a Reply

Panorama Theme by Themocracy