Nexentastor Community EditionでFCストレージ
XenServerおよびESXiで使用できるFCストレージの簡単（？）な作り方
cloudstackを使う上で欠かせない共有ディスク
iSCSI接続は比較的よく見るので、FC接続ストレージを作ってみました。
(1) 準備編
PCI-Express x8が使用できるPCまたはサーバ
Qlogic製Fibre Channel Adapter (QLE2460/QLE2462など)
Intel製ネットワークアダプタ (9301CT等で可)
HDDたくさん
メモリたくさん
私は以下のような環境を準備しました。
M/B：ML110 G6
MEM：16GB
HBA：Qlogic QLE2462 DualPort 4Gbps Fibre Channel Adapter
HDD：Seagate ST2000DM001×8 (LSILogic 9260-8i接続)
WD20EARX×5 (チップセットに接続)
NIC：Intel 9301CT
SSD：Plextor PX-128M5P (L2ARCで使用)
(2) Nexentastor Community Editionのインストール
インストール用にLSILogic 9260-8iで10GBのLUを切り出し、OSをインストール
手順は省略します。。。
(3) FCストレージとしてのセットアップ
sshでNexentastor Community Editionに接続し、セットアップしていきます。
以下でエキスパートモードになります
$ option expert_mode=1 -s
$ !bash
Unix (「raw」) シェルを入力して低レベルの Unix コマンドを実行しようとしています。 警告: 低レベルの Unix コマンドの使用は推奨されません! 実行しますか? Yes
ZFS ストレージプールの作成と確認
# zpool create -f fc-vol00 c1t1d0
# zpool create -f fc-vol01 raidz c2t0d0 c2t1d0 c2t2d0 c2t3d0 c2t4d0
ZFS ストレージプールの確認
# zpool list
NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT
fc-vol00 10.9T 548K 10.9T 0% 1.00x ONLINE -
fc-vol01 9.06T 1.17M 9.06T 0% 1.00x ONLINE -
syspool 19.9G 12.6G 7.26G 63% 1.00x ONLINE -
LUNとして使用するZFSボリュームの作成と確認
# zfs create -V 10T fc-vol00/lu00
# zfs create -V 10G fc-vol00/lu01
# zfs create -V 10G fc-vol00/lu02
# zfs create -V 10G fc-vol00/lu03
# zfs create -V 100G fc-vol00/lu04
# zfs create -V 100G fc-vol00/lu05
# zfs create -V 100G fc-vol00/lu06
# zfs create -V 6144G fc-vol01/lu00
# zfs create -V 10G fc-vol01/lu01
# zfs create -V 10G fc-vol01/lu02
# zfs create -V 10G fc-vol01/lu03
# zfs create -V 100G fc-vol01/lu04
# zfs create -V 100G fc-vol01/lu05
# zfs list
NAME USED AVAIL REFER MOUNTPOINT
fc-vol00 10.6T 59.1G 144K /fc-vol00
fc-vol00/lu00 10.3T 10.3T 25.5G -
fc-vol00/lu01 10.3G 69.4G 72K -
fc-vol00/lu02 10.3G 69.4G 72K -
fc-vol00/lu03 10.3G 69.4G 72K -
fc-vol00/lu04 103G 162G 72K -
fc-vol00/lu05 103G 162G 72K -
fc-vol00/lu06 103G 162G 72K -
fc-vol01 6.42T 723G 230K /fc-vol01
fc-vol01/lu00 6.19T 6.88T 19.7G -
fc-vol01/lu01 10.3G 733G 115K -
fc-vol01/lu02 10.3G 733G 115K -
fc-vol01/lu03 10.3G 733G 115K -
fc-vol01/lu04 103G 826G 115K -
fc-vol01/lu05 103G 826G 115K -
syspool 13.8G 5.76G 164K legacy
syspool/.nza-reserve 144K 5.86G 144K none
syspool/dump 11.2G 5.76G 11.2G -
syspool/rootfs-nmu-000 1.47G 5.76G 1.04G legacy
syspool/rootfs-nmu-001 92K 5.76G 966M legacy
syspool/swap 1.03G 6.79G 72K -
ZFSボリューム用のLUN作成と確認
# stmfadm create-lu /dev/zvol/rdsk/fc-vol00/lu00
Logical unit created: 600144F0504B830000005295FF6D0001
# stmfadm create-lu /dev/zvol/rdsk/fc-vol00/lu01
Logical unit created: 600144F0504B830000005295FF6F0002
# stmfadm create-lu /dev/zvol/rdsk/fc-vol00/lu02
Logical unit created: 600144F0504B830000005295FF710003
# stmfadm create-lu /dev/zvol/rdsk/fc-vol00/lu03
Logical unit created: 600144F0504B830000005295FF720004
# stmfadm create-lu /dev/zvol/rdsk/fc-vol00/lu04
Logical unit created: 600144F0504B830000005295FF740005
# stmfadm create-lu /dev/zvol/rdsk/fc-vol00/lu05
Logical unit created: 600144F0504B83000000529F10F90001
# stmfadm create-lu /dev/zvol/rdsk/fc-vol00/lu06
Logical unit created: 600144F0504B83000000529F19F00001
# stmfadm create-lu /dev/zvol/rdsk/fc-vol01/lu00
Logical unit created: 600144F0504B83000000529B22120001
# stmfadm create-lu /dev/zvol/rdsk/fc-vol01/lu01
Logical unit created: 600144F0504B83000000529F110C0002
# stmfadm create-lu /dev/zvol/rdsk/fc-vol01/lu02
Logical unit created: 600144F0504B83000000529F110F0003
# stmfadm create-lu /dev/zvol/rdsk/fc-vol01/lu03
Logical unit created: 600144F0504B83000000529F11100004
# stmfadm create-lu /dev/zvol/rdsk/fc-vol01/lu04
Logical unit created: 600144F0504B83000000529F11110005
# stmfadm create-lu /dev/zvol/rdsk/fc-vol01/lu05
Logical unit created: 600144F0504B83000000529F11130006
# stmfadm list-lu
LU Name: 600144F0504B830000005295FF6D0001
LU Name: 600144F0504B830000005295FF6F0002
LU Name: 600144F0504B830000005295FF710003
LU Name: 600144F0504B830000005295FF720004
LU Name: 600144F0504B830000005295FF740005
LU Name: 600144F0504B83000000529B22120001
LU Name: 600144F0504B83000000529F10F90001
LU Name: 600144F0504B83000000529F110C0002
LU Name: 600144F0504B83000000529F110F0003
LU Name: 600144F0504B83000000529F11100004
LU Name: 600144F0504B83000000529F11110005
LU Name: 600144F0504B83000000529F11130006
LU Name: 600144F0504B83000000529F19F00001
ホストグループ作成
# stmfadm create-hg esxi01
# stmfadm create-hg esxi02
# stmfadm create-hg vcenter
ホストグループにWWN登録
wwnはストレージに接続するXenServer/ESXiホストのFCアダプタを指定します。
# stmfadm add-hg-member -g esxi01 wwn.10000000c9695686
# stmfadm add-hg-member -g esxi01 wwn.10000000c96955c0
# stmfadm add-hg-member -g esxi02 wwn.10000000c976c1c0
# stmfadm add-hg-member -g esxi02 wwn.10000000c9769a8c
# stmfadm add-hg-member -g vcenter wwn.10000000c960ae93
ホストグループの確認
# stmfadm list-hg -v
Host Group: esxi01
Member: wwn.10000000C9695686
Member: wwn.10000000C96955C0
Host Group: esxi02
Member: wwn.10000000C976C1C0
Member: wwn.10000000C9769A8C
Host Group: vcenter
Member: wwn.10000000c960ae93
ホストグループにLUN ビューを追加
※追加したLUNから順にHost-LUNが0～で割り当てられます
# stmfadm add-view --host-group esxi01 600144F0504B830000005295FF6D0001
# stmfadm add-view --host-group esxi01 600144F0504B830000005295FF6F0002
# stmfadm add-view --host-group esxi01 600144F0504B830000005295FF710003
# stmfadm add-view --host-group esxi01 600144F0504B830000005295FF720004
# stmfadm add-view --host-group esxi01 600144F0504B830000005295FF740005
# stmfadm add-view --host-group esxi01 600144F0504B83000000529B22120001
# stmfadm add-view --host-group esxi01 600144F0504B83000000529F10F90001
# stmfadm add-view --host-group esxi01 600144F0504B83000000529F110C0002
# stmfadm add-view --host-group esxi01 600144F0504B83000000529F110F0003
# stmfadm add-view --host-group esxi01 600144F0504B83000000529F11100004
# stmfadm add-view --host-group esxi01 600144F0504B83000000529F11110005
# stmfadm add-view --host-group esxi01 600144F0504B83000000529F11130006
# stmfadm add-view --host-group esxi02 600144F0504B830000005295FF6D0001
# stmfadm add-view --host-group esxi02 600144F0504B830000005295FF6F0002
# stmfadm add-view --host-group esxi02 600144F0504B830000005295FF710003
# stmfadm add-view --host-group esxi02 600144F0504B830000005295FF720004
# stmfadm add-view --host-group esxi02 600144F0504B830000005295FF740005
# stmfadm add-view --host-group esxi02 600144F0504B83000000529B22120001
# stmfadm add-view --host-group esxi02 600144F0504B83000000529F10F90001
# stmfadm add-view --host-group esxi02 600144F0504B83000000529F110C0002
# stmfadm add-view --host-group esxi02 600144F0504B83000000529F110F0003
# stmfadm add-view --host-group esxi02 600144F0504B83000000529F11100004
# stmfadm add-view --host-group esxi02 600144F0504B83000000529F11110005
# stmfadm add-view --host-group esxi02 600144F0504B83000000529F11130006
# stmfadm add-view --host-group vcenter 600144F0504B83000000529F19F00001
LUN の構成を確認
# stmfadm list-view -l 600144F0504B830000005295FF6D0001
View Entry: 0
Host group : esxi01
Target group : All
LUN : 0
View Entry: 1
Host group : esxi02
Target group : All
LUN : 0
# stmfadm list-view -l 600144F0504B830000005295FF6F0002
View Entry: 0
Host group : esxi01
Target group : All
LUN : 1
View Entry: 1
Host group : esxi02
Target group : All
LUN : 1
# stmfadm list-view -l 600144F0504B830000005295FF710003
View Entry: 0
Host group : esxi01
Target group : All
LUN : 2
View Entry: 1
Host group : esxi02
Target group : All
LUN : 2
# stmfadm list-view -l 600144F0504B830000005295FF720004
View Entry: 0
Host group : esxi01
Target group : All
LUN : 3
View Entry: 1
Host group : esxi02
Target group : All
LUN : 3
# stmfadm list-view -l 600144F0504B830000005295FF740005
View Entry: 0
Host group : esxi01
Target group : All
LUN : 4
View Entry: 1
Host group : esxi02
Target group : All
LUN : 4
# stmfadm list-view -l 600144F0504B83000000529B22120001
View Entry: 0
Host group : esxi01
Target group : All
LUN : 5
View Entry: 1
Host group : esxi02
Target group : All
LUN : 5
# stmfadm list-view -l 600144F0504B83000000529F10F90001
View Entry: 0
Host group : esxi01
Target group : All
LUN : 6
View Entry: 1
Host group : esxi02
Target group : All
LUN : 6
# stmfadm list-view -l 600144F0504B83000000529F110C0002
View Entry: 0
Host group : esxi01
Target group : All
LUN : 7
View Entry: 1
Host group : esxi02
Target group : All
LUN : 7
# stmfadm list-view -l 600144F0504B83000000529F110F0003
View Entry: 0
Host group : esxi01
Target group : All
LUN : 8
View Entry: 1
Host group : esxi02
Target group : All
LUN : 8
# stmfadm list-view -l 600144F0504B83000000529F11100004
View Entry: 0
Host group : esxi01
Target group : All
LUN : 9
View Entry: 1
Host group : esxi02
Target group : All
LUN : 9
# stmfadm list-view -l 600144F0504B83000000529F11110005
View Entry: 0
Host group : esxi01
Target group : All
LUN : 10
View Entry: 1
Host group : esxi02
Target group : All
LUN : 10
# stmfadm list-view -l 600144F0504B83000000529F11130006
View Entry: 0
Host group : esxi01
Target group : All
LUN : 11
View Entry: 1
Host group : esxi02
Target group : All
LUN : 11
# stmfadm list-view -l 600144F0504B83000000529F19F00001
View Entry: 0
Host group : vcenter
Target group : All
LUN : 0
FCアダプタドライバのモードを確認します
# mdb -k
Loading modules: [ unix genunix specfs dtrace mac cpu.generic uppc pcplusmp scsi_vhci zfs mr_sas sd sockfs ip hook neti sctp arp usba qlc stmf_sbd stmf lofs idm random nfs crypto sata cpc smbsrv logindmux ptm nsctl sdbc ufs nsmb sv rdc ii ]
> ::devbindings -q qlc
ffffff04e606c020 pciex1077,2432, instance #0 (driver name: qlc)
ffffff04e606a7c0 pciex1077,2432, instance #1 (driver name: qlc)
> $q
全てのFCポートをターゲットモードに変更する
現在のバインドを削除する
# update_drv -d -i 'pciex1077,2432' qlc
Cannot unload module: qlc
Will be unloaded upon reboot.
新しいバインドを確立する
# update_drv -a -i 'pciex1077,2432' qlt
devfsadm: driver failed to attach: qlt
Warning: Driver (qlt) successfully added to system but failed to attach
システムをリブートして新しいドライバをロードする
# init 6
ターゲットモードフレームワークが HBA ポートにアクセスできることを確認
# stmfadm list-target -v
Target: wwn.210000E08B854AE2
Operational Status: Online
Provider Name : qlt
Alias : qlt1,0
Protocol : Fibre Channel
Sessions : 1
Initiator: wwn.10000000C976C1C0
Alias: Emulex LPe1150-F4-N FV2.72A2 DV10.0.100.1
Logged in since: Wed Nov 27 23:35:22 2013
Target: wwn.210000E08B8557DE
Operational Status: Online
Provider Name : qlt
Alias : qlt0,0
Protocol : Fibre Channel
Sessions : 1
Initiator: wwn.10000000C9769A8C
Alias: Emulex LPe1150-F4-N FV2.72A2 DV10.0.100.1
Logged in since: Wed Nov 27 23:35:21 2013
ESXiから認識させた例
LUNが細かく分かれてるのは・・・RDMで使うからですハイ。
