goofysでS3をマウントしてみる

dorako321

9年前

以前、AWSのS3をEC2にマウントするs3fsを試してみるという記事を書きましたがgoofysを使えばs3fsより高速にファイルを転送できるとのことでしたのでこちらもためしてみることに。

前提パッケージのインストール

sudo yum install golang fuse -y

1	sudo yum install golang fuse -y

goofysのインストール

goはさっぱり…

export GOPATH=$HOME/go
go get github.com/kahing/goofys
go install github.com/kahing/goofys
$GOPATH/bin/goofys -h

Error: goofys takes exactly two arguments.

NAME:
 goofys - Mount an S3 bucket locally

USAGE:
 goofys [global options] bucket mountpoint

VERSION:
 0.0.5

GLOBAL OPTIONS:
 --help, -h Print this help text and exit successfuly.
 -o [-o option -o option] Additional system-specific mount options. Be careful!
 --dir-mode "493" Permissions bits for directories. (default: 0755)
 --file-mode "420" Permission bits for files (default: 0644)
 --uid "500" UID owner of all inodes.
 --gid "500" GID owner of all inodes.
 --endpoint The non-AWS endpoint to connect to. Possible values: http://127.0.0.1:8081/
 --region "us-west-2" The non-AWS endpoint to connect to. Possible values: us-east-1, us-west-1, us-west-2, eu-west-1, eu-central-1, ap-southeast-1, ap-southeast-2, ap-northeast-1, sa-east-1, cn-north-1
 --storage-class "STANDARD" The type of storage to use when writing objects. Possible values: REDUCED_REDUNDANCY, STANDARD (default), STANDARD_IA.
 --use-path-request Use a path-style request instead of virtual host-style. (deprecated, always on)
 --profile Use a named profile from $HOME/.aws/credentials instead of "default"
 --stat-cache-ttl "1m0s" How long to cache StatObject results and inode attributes.
 --type-cache-ttl "1m0s" How long to cache name -> file/dir mappings in directory inodes.
 --debug_fuse Enable fuse-related debugging output.
 --debug_s3 Enable S3-related debugging output.
 -f Run goofys in foreground.
 --version, -v print the version

export GOPATH=$HOME/go

go get github.com/kahing/goofys

go install github.com/kahing/goofys

$GOPATH/bin/goofys -h

Error: goofys takes exactly two arguments.

NAME:

goofys - Mount an S3 bucket locally

USAGE:

goofys [global options] bucket mountpoint

VERSION:

0.0.5

GLOBAL OPTIONS:

--help, -h Print this help text and exit successfuly.

-o [-o option -o option] Additional system-specific mount options. Be careful!

--dir-mode "493" Permissions bits for directories. (default: 0755)

--file-mode "420" Permission bits for files (default: 0644)

--uid "500" UID owner of all inodes.

--gid "500" GID owner of all inodes.

--endpoint The non-AWS endpoint to connect to. Possible values: http://127.0.0.1:8081/

--region "us-west-2" The non-AWS endpoint to connect to. Possible values: us-east-1, us-west-1, us-west-2, eu-west-1, eu-central-1, ap-southeast-1, ap-southeast-2, ap-northeast-1, sa-east-1, cn-north-1

--storage-class "STANDARD" The type of storage to use when writing objects. Possible values: REDUCED_REDUNDANCY, STANDARD (default), STANDARD_IA.

--use-path-request Use a path-style request instead of virtual host-style. (deprecated, always on)

--profile Use a named profile from $HOME/.aws/credentials instead of "default"

--stat-cache-ttl "1m0s" How long to cache StatObject results and inode attributes.

--type-cache-ttl "1m0s" How long to cache name -> file/dir mappings in directory inodes.

--debug_fuse Enable fuse-related debugging output.

--debug_s3 Enable S3-related debugging output.

-f Run goofys in foreground.

--version, -v print the version

バケットのマウント

$ mkdir ~/goofys
$ $GOPATH/bin/goofys s3dj-extream ~/goofys
$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/xvda1      7.8G  1.6G  6.2G  20% /
devtmpfs        238M   56K  238M   1% /dev
tmpfs           246M     0  246M   0% /dev/shm
s3dj-extream    1.0P     0  1.0P   0% /home/ec2-user/goofys

$ mkdir ~/goofys

$ $GOPATH/bin/goofys s3dj-extream ~/goofys

$ df -h

Filesystem Size Used Avail Use% Mounted on

/dev/xvda1 7.8G 1.6G 6.2G 20% /

devtmpfs 238M 56K 238M 1% /dev

tmpfs 246M 0 246M 0% /dev/shm

s3dj-extream 1.0P 0 1.0P 0% /home/ec2-user/goofys

あれs3fsの時は256テラバイトだったのに、1ペタバイトになってる。わお!

$ cd goofys/
$ touch fuga
$ ls
fuga  hoge

$ cd goofys/

$ touch fuga

$ ls

fuga hoge

hogeはs3fsで試した時のデータです。

速度を比較してみる

s3fs

$ i=1; time while [ $i -le 500 ]
> do
> touch s3fs_test_$i > /dev/null 2>&1
> i=$((i+1))
> done

real 2m15.855s
user 0m0.040s
sys 0m0.068s

$ i=1; time while [ $i -le 500 ]

> do

> touch s3fs_test_$i > /dev/null 2>&1

> i=$((i+1))

> done

real 2m15.855s

user 0m0.040s

sys 0m0.068s

あれ?classmethodさんの記事の結果よりさらに遅くなってます。EC2のスペックも影響してそうですね。

goofys

$ i=1; time while [ $i -le 500 ]
> do
>  touch goofys_test_$i > /dev/null 2>&1
>  i=$((i+1))
> done

real    0m26.767s
user    0m0.080s
sys     0m0.016s

$ i=1; time while [ $i -le 500 ]

> do

> touch goofys_test_$i > /dev/null 2>&1

> i=$((i+1))

> done

real 0m26.767s

user 0m0.080s

sys 0m0.016s

goofysはなるほど耐えられるレベルではあります。今使うならgoofysでしょうか。参考にした記事から日がたってるので改善されているか期待しましたが、そんなことはなかったようです。