All Articles

AWS Batch๋กœ Gensim ๋Œ๋ฆฌ๊ธฐ

Gensim์„ ๋Œ๋ฆฌ๋‹ค๋ณด๋ฉด ๋žจ์ด ๋ชจ์ž๋ž„ ๋•Œ๊ฐ€ ๋งŽ์•„์„œ EC2๋ฅผ ์ž ๊น ๋„์›Œ์„œ ๋Œ๋ ธ์—ˆ๋Š”๋ฐ ์ด๊ฒƒ๋„ ํ•˜๋‹ค๋ณด๋‹ˆ๊นŒ ๋ฐ˜๋ณต์ธ ๊ฒƒ ๊ฐ™์•„์„œ AWS Batch๋ฅผ ์จ๋ณด๊ธฐ๋กœ ํ–ˆ๋‹ค. ๊ธฐ๋ณธ์ ์œผ๋กœ๋Š” ์˜ˆ์ œ๋กœ ๋‚˜์˜ค๋Š” Fetch & Run์ด๋ž‘ ํฌ๊ฒŒ ๋‹ค๋ฅผ๊ฒŒ ์—†๋Š”๋ฐ ํƒœ์Šคํฌ๋ฅผ ๋Œ๋ฆด ์ด๋ฏธ์ง€๋ฅผ ๋งŒ๋“ค ๋•Œ ์‹ค์ˆ˜ํ•˜๊ธฐ ์‰ฌ์šด ๋ถ€๋ถ„์ด ์žˆ๋‹ค. ์‚ฌ์‹ค ์Šคํฌ๋ฆฝํŠธ๋กœ EC2 ๋„์šฐ๋Š” ๊ฑฐ๋ž‘ ํฌ๊ฒŒ ์—†์ง€๋งŒ ์š”๊ตฌํ•˜๋Š” ์‚ฌ์–‘์— ๋งž์ถ”์–ด์„œ ์ ์ ˆํ•˜๊ฒŒ ์ŠคํŒŸ ์ธ์Šคํ„ด์Šค๋กœ ๋Œ๋ ค์ฃผ๊ณ , ๋˜ ๋„์šฐ๊ณ  ๋‚ด๋ฆฌ๋Š” ๊ณผ์ •์ด ์ž๋™์ด๋ผ์„œ ์ด์ฐธ์— ์จ๋ณด๊ธฐ๋กœ ํ–ˆ๋‹ค. ๋‹ค์Œ์€ ์ด๋ฏธ์ง€๋ฅผ ๋งŒ๋“ค ๋•Œ ์œ ์˜ํ•  ์ .

  • amazonlinux ์ด๋ฏธ์ง€ ์œ„์—๋‹ค๊ฐ€ ๋งŒ๋“ค ๊ฒƒ. ์ฒ˜์Œ์—๋Š” python:3.6 ์ด๋ฏธ์ง€๋กœ ํ–ˆ๋‹ค๊ฐ€ ๋‚˜์ค‘์— CloudWatch์—์„œ ๋กœ๊ทธ๋ฅผ ํ™•์ธํ•˜๊ธฐ ์œ„ํ•ด์„œ awslogs๋ฅผ ๊น”์•„์•ผ ํ•จ์„ ๊นจ๋‹ซ๊ณ , ๊น”๋ ค๋‹ค๊ฐ€, ์ธ์Šคํ†จ๋Ÿฌ๋Š” ํŒŒ์ด์ฌ 3.6์„ ์ง€์› ์•ˆํ•˜๊ณ โ€ฆ๊ทธ๋ƒฅ ์•„๋งˆ์กด ๋ฆฌ๋ˆ…์Šค์—์„œ ๊นŒ๋Š”๊ฒŒ ๋” ๋‚ซ๊ฒ ๋‹ค ํ•ด์„œ ๋ฒ ์ด์Šค ์ด๋ฏธ์ง€๋ฅผ ๋ฐ”๊ฟจ๋‹ค.
  • gensim์„ ๊น”๊ธฐ ์ „์— ์ปดํŒŒ์ผ๋Ÿฌ๋“ค์ด๋ž‘ python-devel๋„ ๊ฐ™์ด ๊น”์•„์ค„ ๊ฒƒ. ์ด๊ฑฐ ์—†์œผ๋ฉด C์ปดํŒŒ์ผ์„ ์•ˆํ•ด์„œ FAST_VERSION ๊ฐ’์ด 1๋กœ ์•ˆ๋‚˜์˜ค๊ณ , ํŠธ๋ ˆ์ด๋‹์ด ๋Š๋ฆด๊บผ๋ผ๊ณ  ๊ฒฝ๊ณ ๊ฐ€ ๋‚˜์˜จ๋‹ค. python:3.6 ์ด๋ฏธ์ง€์—์„œ๋Š” ๊ทธ๋ƒฅ ์ž˜ ๋˜์—ˆ๋Š”๋ฐ amazonlinux์—์„œ๋Š” ์„ค์น˜๊ฐ€ ํ•„์š”ํ–ˆ๋‹ค.
  • ์ด๊ฒƒ๋งŒ ์ฃผ์˜ํ•˜๋ฉด ๊ทธ๋ƒฅ ๋ฆฌ๋ˆ…์Šค๋‹ค. ์‚ฝ์งˆ ๋์— ์ •์ฐฉํ•œ Dockerfile์€ ๋‹ค์Œ๊ณผ ๊ฐ™๋‹ค.
# Dockerfile
FROM amazonlinux:latest

# Ubuntu์—์„œ build-essential ๋น„์Šทํ•œ ๊ฒƒ
RUN yum groupinstall -y "Development Tools"

RUN yum install -y python3
RUN yum install -y python3-devel

RUN pip3 install gensim
RUN pip3 install awscli
RUN python3 check_fast_version.py
# check_fast_version.py
import sys

try:
    from gensim.models.word2vec_inner import FAST_VERSION

    print('FAST_VERSION ok ! Retrieved with value ', FAST_VERSION)
    sys.exit()
except ImportError:
    print('Failed... fall back to plain numpy (20-80x slower training than the above)')
    sys.exit(-1)

๋Š๋‚€์ 

  • ์• ์ดˆ์— ๊ฐœ๋ฐœ์„ ๋„์ปค ์œ„์—์„œ ํ•ด์•ผ์ง€ ๋กœ์ปฌ์—์„œ ํ•ด๋ณด๊ณ  ์˜ฎ๊ธฐ๋ ค๊ณ  ํ•˜๋ฉด ์‹œ๊ฐ„์„ ๋„ˆ๋ฌด ๋งŽ์ด ์“ด๋‹ค. ์–ด๋–ค ์„ ๋ฐฐ ๋ธ”๋กœ๊ทธ์—์„œ ๋ชจ๋“  ๋กœ์ปฌ ๊ฐœ๋ฐœ ํ™˜๊ฒฝ์„ ๋„์ปค๋กœ๋งŒ ๊ตฌ์„ฑํ•œ๋‹ค๊ณ  ์ฝ์—ˆ๋Š”๋ฐ ๋ฌด์Šจ ์˜๋ฏธ์ธ์ง€ ์•Œ๊ฒ ๋‹ค.
  • s3์—์„œ ๋ฐ์ดํ„ฐ๋ฅผ ๊ฐ€์ ธ์˜ค๋Š” ๊ฒฝ์šฐ์—๋Š” ์• ์ดˆ์— ๊ฐœ๋ฐœ์„ AWS ์ธ์Šคํ„ด์Šค ์œ„์—์„œ ํ•ด๋„ ์ข‹๊ฒ ๋‹ค.
  • ๋กœ์ปฌ์—์„œ ๋žจ์ด ๋ชจ์ž๋ผ์„œ ์•ˆ ๋Œ์•„๊ฐ€๋˜๊ฑฐ๋„ ๊ทธ๋ƒฅ ์ฝ”์–ด / ์šฉ๋Ÿ‰์„ ํŒจ๋Ÿฌ๋ฏธํ„ฐ๋กœ ๋„ฃ์–ด ์ฃผ๋ฉด ์•Œ์•„์„œ ์ธ์Šคํ„ด์Šค๊ฐ€ ์ƒ๊ธฐ๊ณ , ์ผ์ด ๋๋‚˜๋ฉด ๋‹ค์‹œ ๊บผ์ง€๋‹ˆ๊นŒ ๋„ˆ๋ฌด ํŽธํ•˜๋‹ค +_+
Published 28 Oct 2018

If I keep marking the dots, someday they will ๐Ÿ”—๐Ÿ”—
Hyeungshik Jung on Twitter