Fluentd meetup #3 LT資料

最近ずーっと仕事で使わせていただいていて個人的になじみ深いOSSとなったFluentdですが、先日IIJ様でFluentd meetup #3が開催されましたので行ってきました。

今回のイベントではLTをさせていただきましたので資料を公開します。

LT資料

補足

現状はあくまで独自に拙訳しただけですので、引き続きFluentd Projectの日本語版作成に関するissueでより良い翻訳・公開の方法を相談させていただきたいと考えています。

資料にも記載していますが…

また、誤訳や日本語の言い回しで気になる部分がありましたらGitHubのissueなどでお知らせいただけるとうれしいです。

f:id:mazgi:20121111125745j:plain

主催者の皆様、すばらしい会場と機材を提供してくださったIIJ様、参加者の皆様、 ありがとうございました!

ローカル関数便利♪

こーゆーことができるらしい。

def func1(s1:String):String={
  def func2(s2:String):String={
    def func3(s3:String):String={
      return s3+"3"
    }
    return func3(s2+"2")
  }
  return func2(s1+"1")
}

実行してみる。

scala> def func1(s1:String):String={
     |   def func2(s2:String):String={
     |     def func3(s3:String):String={
     |       return s3+"3"
     |     }
     |     return func3(s2+"2")
     |   }
     |   return func2(s1+"1")
     | }
func1: (String)String

scala> println(func1("hoge"))
hoge123

GentooとOSXでiSCSI♪

ベタにiscsitargetとglobalSAN iSCSIで。

設定とかはさておき、ベンチマーク。
ふむふむ、Xbenchってのがあるのか・・・(・0・。)

ちなみに、Target側のHDDはUDMA5で、ネットワークは間に100MbpsなHUBが入ってますorz

Local HDD

Results 32.60
System Info
Xbench Version 1.3
System Version 10.4.11 (8S2167)
Physical RAM 4096 MB
Model iMac5,1
Drive Type ST3160812AS Q
Disk Test 32.60
Sequential 60.68
Uncached Write 53.11 32.61 MB/sec [4K blocks]
Uncached Write 60.48 34.22 MB/sec [256K blocks]
Uncached Read 65.11 19.05 MB/sec [4K blocks]
Uncached Read 65.81 33.07 MB/sec [256K blocks]
Random 22.29
Uncached Write 7.64 0.81 MB/sec [4K blocks]
Uncached Write 56.86 18.20 MB/sec [256K blocks]
Uncached Read 54.35 0.39 MB/sec [4K blocks]
Uncached Read 79.68 14.79 MB/sec [256K blocks]

iSCSI

Results 25.02
System Info
Xbench Version 1.3
System Version 10.4.11 (8S2167)
Physical RAM 4096 MB
Model iMac5,1
Drive Type IET VIRTUAL-DISK
Disk Test 25.02
Sequential 18.35
Uncached Write 18.26 11.21 MB/sec [4K blocks]
Uncached Write 17.62 9.97 MB/sec [256K blocks]
Uncached Read 16.87 4.94 MB/sec [4K blocks]
Uncached Read 21.22 10.66 MB/sec [256K blocks]
Random 39.30
Uncached Write 19.62 2.08 MB/sec [4K blocks]
Uncached Write 31.42 10.06 MB/sec [256K blocks]
Uncached Read 661.55 4.69 MB/sec [4K blocks]
Uncached Read 57.24 10.62 MB/sec [256K blocks]

やっぱりついでに、/etc/ietd.conf

Target iqn.2008-04.com.HUNYA:strage.disk0.4iMac
Lun 0 Path=/dev/hda4,Type=fileio
IncomingUser NYORO

LVM設定 -LV作成-

全部ReiserFSで。
オンラインで拡張できるから2GBでいいや。
# for p in Home Var Portage Tmp Opt; do lvcreate -n$p -L2G VG0; done
# for p in /dev/VG0/*; do mkreiserfs $p; done

結果はこんな感じ
# lvdisplay
--- Logical volume ---
LV Name /dev/VG0/Home
VG Name VG0
LV UUID PSpzkE-AmSp-EweF-f8lt-haH4-nZyK-L6mj9c
LV Write Access read/write
LV Status available
# open 0
LV Size 2.00 GB
Current LE 512
Segments 1
Allocation inherit
Read ahead sectors 0
Block device 253:0

--- Logical volume ---
LV Name /dev/VG0/Var
VG Name VG0
LV UUID YqA1cx-B72h-K71V-Cor4-h6AX-qBO6-Q3I20K
LV Write Access read/write
LV Status available
# open 0
LV Size 2.00 GB
Current LE 512
Segments 1
Allocation inherit
Read ahead sectors 0
Block device 253:1

--- Logical volume ---
LV Name /dev/VG0/Portage
VG Name VG0
LV UUID CICmO3-3Hnm-1K9e-gAfQ-aGRT-6F1f-XRw0aB
LV Write Access read/write
LV Status available
# open 0
LV Size 2.00 GB
Current LE 512
Segments 1
Allocation inherit
Read ahead sectors 0
Block device 253:2

--- Logical volume ---
LV Name /dev/VG0/Tmp
VG Name VG0
LV UUID nH0nGd-523t-Hr3u-eOod-3swt-KXeD-ALH4lH
LV Write Access read/write
LV Status available
# open 0
LV Size 2.00 GB
Current LE 512
Segments 1
Allocation inherit
Read ahead sectors 0
Block device 253:3

--- Logical volume ---
LV Name /dev/VG0/Opt
VG Name VG0
LV UUID msJKKf-2ISi-AkQ6-y4Ix-oZkb-4vG6-TnZ5UW
LV Write Access read/write
LV Status available
# open 0
LV Size 2.00 GB
Current LE 512
Segments 1
Allocation inherit
Read ahead sectors 0
Block device 253:4

LVM設定 -LVの最大サイズ-

# pvcreate /dev/md0
Physical volume "/dev/md0" successfully created
# vgcreate VG0 /dev/md0
Volume group "VG0" successfully created
livecd ~ # vgdisplay
--- Volume group ---
VG Name VG0
System ID
Format lvm2
Metadata Areas 1
Metadata Sequence No 1
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 0
Open LV 0
Max PV 0
Cur PV 1
Act PV 1
VG Size 298.09 GB
PE Size 4.00 MB
Total PE 76311
Alloc PE / Size 0 / 0
Free PE / Size 76311 / 298.09 GB
VG UUID Lrn8oM-PDa1-U0Ca-P2Er-kpZj-2Mnl-Pg6H68

# lvcreate -nLV0 -L298G VG0
Logical volume "LV0" created
# lvdisplay
--- Logical volume ---
LV Name /dev/VG0/LV0
VG Name VG0
LV UUID VkrE36-fp3k-ggi4-2Aj5-Sqao-Ot9m-Wkhwr4
LV Write Access read/write
LV Status available
# open 0
LV Size 298.00 GB
Current LE 76288
Segments 1
Allocation inherit
Read ahead sectors 0
Block device 253:0

あれ?PEx65536が最大値じゃなかった?


これでどーだ!?
# vgcreate -s8K VG0 /dev/md0
Volume group "VG0" successfully created
# vgdisplay
--- Volume group ---
VG Name VG0
System ID
Format lvm2
Metadata Areas 1
Metadata Sequence No 1
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 0
Open LV 0
Max PV 0
Cur PV 1
Act PV 1
VG Size 298.09 GB
PE Size 8.00 KB
Total PE 39071368
Alloc PE / Size 0 / 0
Free PE / Size 39071368 / 298.09 GB
VG UUID QN8MMr-5sQ4-IxyO-3fcG-USFd-PJip-yKqTOb

# lvcreate -nLV0 -L298G VG0
Logical volume "LV0" created
# lvdisplay
--- Logical volume ---
LV Name /dev/VG0/LV0
VG Name VG0
LV UUID XZcblP-oXaw-OK02-Xq3P-URT9-riQA-4oLIPH
LV Write Access read/write
LV Status available
# open 0
LV Size 298.00 GB
Current LE 39059456
Segments 1
Allocation inherit
Read ahead sectors 0
Block device 253:0

# vgdisplay
--- Volume group ---
VG Name VG0
System ID
Format lvm2
Metadata Areas 1
Metadata Sequence No 2
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 1
Open LV 0
Max PV 0
Cur PV 1
Act PV 1
VG Size 298.09 GB
PE Size 8.00 KB
Total PE 39071368
Alloc PE / Size 39059456 / 298.00 GB
Free PE / Size 11912 / 93.06 MB
VG UUID QN8MMr-5sQ4-IxyO-3fcG-USFd-PJip-yKqTOb

うぬ・・・
とりあえずPEの最大数とか気にしなくていいらしい。。

ソフトウェアRAID構築

結局これに落ち着くわけで。

/dev/sd[ab]をRAID1で
# mdadm --create /dev/md0 --level=1 --raid-devices=2 /dev/sda /dev/sdb

途中経過はこんな感じ。
Every 2.0s: cat /proc/mdstat Sat Apr 5 01:09:02 2008

Personalities : [raid1]
md0 : active raid1 sdb[1] sda[0]
312571136 blocks [2/2] [UU]
[=>...................] resync = 6.8% (21260160/312571136) finish=60.1mi
n speed=80701K/sec

unused devices:

できた♪
Every 2.0s: cat /proc/mdstat Sat Apr 5 02:44:38 2008

Personalities : [raid1]
md0 : active raid1 sdb[1] sda[0]
312571136 blocks [2/2] [UU]

unused devices:

LVMでMirrorして片方外してみる

# mkreiserfs /dev/vg/lvol0
# mkdir /mnt/disk1
# mount /dev/vg/lvol0 /mnt/disk1/
# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 115G 2.7G 107G 3% /
udev 10M 216K 9.8M 3% /dev
none 251M 0 251M 0% /dev/shm
/dev/mapper/vg-lvol0 10G 33M 10G 1% /mnt/disk1

# cd /mnt/disk1/
# od -x /dev/random > random.txt
# md5sum random.txt > random.md5
# ls -l
total 4

  • rw-r--r-- 1 root root 45 Mar 21 23:55 random.md5
  • rw-r--r-- 1 root root 0 Mar 21 23:55 random.txt

# reboot
(HDD1台取り外し)

# vgdisplay
Couldn't find device with uuid 'VDCbqp-oWz7-ffD8-abb2-kM2y-Asdq-B3XLyG'.
Couldn't find all physical volumes for volume group vg.
Couldn't find device with uuid 'VDCbqp-oWz7-ffD8-abb2-kM2y-Asdq-B3XLyG'.
Couldn't find all physical volumes for volume group vg.
Volume group "vg" not found
# pvdisplay
Couldn't find device with uuid 'VDCbqp-oWz7-ffD8-abb2-kM2y-Asdq-B3XLyG'.
Couldn't find device with uuid 'VDCbqp-oWz7-ffD8-abb2-kM2y-Asdq-B3XLyG'.
Couldn't find device with uuid 'VDCbqp-oWz7-ffD8-abb2-kM2y-Asdq-B3XLyG'.
Couldn't find all physical volumes for volume group vg.
Couldn't find device with uuid 'VDCbqp-oWz7-ffD8-abb2-kM2y-Asdq-B3XLyG'.
Couldn't find all physical volumes for volume group vg.
Can't read vg: skipping
Couldn't find device with uuid 'VDCbqp-oWz7-ffD8-abb2-kM2y-Asdq-B3XLyG'.
Couldn't find all physical volumes for volume group vg.
Couldn't find device with uuid 'VDCbqp-oWz7-ffD8-abb2-kM2y-Asdq-B3XLyG'.
Couldn't find all physical volumes for volume group vg.
Can't read vg: skipping

・・・ダメじゃん。。。
そもそもソフトウェアRAIDの代替になるような機能ではない?
よくわかりません>_