All Articles

Evolution of web-scale engagement modeling at Pinterest ์„ NVIDA์—์„œ ํ•œ ํ–‰์‚ฌ์—์„œ ๋‚˜์˜จ ํ†ก์ธ๋ฐ ์ž๊ธฐ๋„ค๋“ค์ด ํ•ด์˜จ ๊ฒƒ๋“ค ๊ทธ๋ฆฌ๊ณ  ์š”์ƒˆ ํ•˜๋Š” ๊ฒƒ๋“ค์„ ์†Œ์ƒํžˆ ์ ์–ด์ฃผ๊ณ  ์žˆ์–ด์„œ ์žฌ๋ฐŒ๊ฒŒ ๋ณด์•˜๋‹ค. ๊ฒฐ๊ตญ ์ด ํ†ก์€ ์ด ๊ทธ๋ฆผ ํ•œ ์žฅ์œผ๋กœ ์š”์•ฝ์ด ๋œ๋‹ค. evoluion ์‚ฌ์‹ค ํ†ก์„ ๋ณด๋ฉด ์ œ์ผ ์ข‹์€๋ฐ ์ด ํฌ์ŠคํŒ…์„ ๋ณด์‹œ๋Š” ๋ถ„๋“ค์€ ๋ณผ๊นŒ ๋ง๊นŒ ๊ณ ๋ฏผํ•˜์‹œ๋Š” ๋ถ„์ผ ํ™•๋ฅ ์ด ๋†’์œผ๋‹ˆ ๊ฐ„๋‹จํ•˜๊ฒŒ ์ •๋ฆฌํ•ด๋ณธ๋‹ค.


  • First ML model
    • ์ฒ˜์Œ์œผ๋กœ ML ๋ชจ๋ธ์„ ๋„์ž…ํ–ˆ๊ณ  ๊ฐ„๋‹จํ•œ logistic regression ๋ชจ๋ธ์ด์—ˆ๋‹ค๊ณ  ํ•œ๋‹ค
  • GBDT model
    • GBDT ๋ชจ๋ธ์„ ๋„์ž…ํ•˜์…”์„œ ์ด์ „ ๋ชจ๋ธ์— ๋น„ํ•ด ํšจ๊ณผ๋ฅผ ๋งŽ์ด ๋ณด์…จ๋‹ค๊ณ 
  • NN model
    • ์ฒ˜์Œ์œผ๋กœ NN๋ชจ๋ธ ์‹œ์ž‘.
  • Deep Multitask NN model
  • Realtime Features
    • ์‰ฝ๊ฒŒ ์–˜๊ธฐํ•ด์„œ ๋ฐฉ๊ธˆ ๋ญ˜ ๋ˆŒ๋ €๋Š”์ง€๊ฐ€ ๋ฐ”๋กœ ํ”ผ์ฒ˜๋กœ ๋“ค์–ด๊ฐ€์•ผ ํ•œ๋‹ค๋Š” ์ด์•ผ๊ธฐ
  • PinSAGE
    • ์„œ๋น™ ํƒ€์ž„์— ๋ชจ๋ธ์—๋‹ค๊ฐ€ ์œ ์ €/ํ•€์˜ ๋ชจ๋“  ํžˆ์Šคํ† ๋ฆญ ๋ฐ์ดํ„ฐ๋ฅผ ๋„ฃ์„ ์ˆ˜ ์—ˆ์œผ๋‹ˆ ์˜คํ”„๋ผ์ธ์—์„œ ๋ฏธ๋ฆฌ ์ž„๋ฒ ๋”ฉ์— ๋…น์—ฌ๋†“๊ณ  ๋ชจ๋ธ์—๋Š” ์ด๊ฒƒ๋งŒ ๋„ฃ๋Š” ๊ฒƒ์ด ํšจ์œจ์ ์ด๋‹ค.
    • PinSAGE๋Š” Graph Neural Network๋ฅผ ์ด์šฉํ•œ ์ƒํ’ˆ ์ž„๋ฒ ๋”ฉ
    • PinnerFormer๋Š” (์ด๋ฆ„ ์ฐธ ์ž˜ ์ง€์—ˆ๋‹ค) Transformer๋ฅผ ์ด์šฉํ•œ ์œ ์ € ์ž„๋ฒ ๋”ฉ
  • Offline Exp Speedup
    • ๋‹จ์ˆœํ•œ mAP ์ด๋Ÿฐ๊ฑฐ ๋ง๊ณ  ์˜คํ”„๋ผ์ธ์œผ๋กœ ๋” ์‹คํ—˜ ๊ฒฐ๊ณผ๋ฅผ ์ž˜ ์˜ˆ์ธกํ•  ์ˆ˜ ์žˆ๋Š” ํˆด๋ง์ด ์ƒ๊ฒผ๋‹ค๋Š”๋ฐ ์†”์งํžˆ ์ดํ•ด๋ฅผ ์ž˜ ๋ชปํ–ˆ๋‹ค.
    • ํ•ด๋‹น ๋ธ”๋กœ๊ทธ ํฌ์ŠคํŒ… ์ฝ์–ด๋ณด๊ณ  ๋‚ด์šฉ์„ ์—…๋ฐ์ดํŠธ ํ•˜๊ฒ ๋‹ค.
  • Wide Networks
    • ๊ธฐ์กด์—๋Š” Representation / Summarization / Latent Cross ์ด๋Ÿฐ ์‹์œผ๋กœ Fully connected layer ์ „์— ์—ฌ๋Ÿฌ ๋‹จ๊ณ„๊ฐ€ ์žˆ์—ˆ๋Š”๋ฐ, ์œ ์ €/ํ•€ ์ž„๋ฒ ๋”ฉ์„ ์“ฐ๋ฉด ์ด๊ฒŒ ๋น„ํšจ์œจ์ ์ด์–ด์„œ ๊ทธ๋ƒฅ Wide Fully connected layer๋กœ ๋ฐ”๊พธ์…จ๋‹ค๊ณ .
    • ์ด๋Ÿฌ๋ฉด ๋ ˆ์ดํ„ด์‹œ๊ฐ€ ๋Š๋Š”๋ฐ quantization์œผ๋กœ ์žก์œผ์…จ๋‹ค.
  • Transformer
    • DCNv2, PLE ๊ฐ™์€ ๋ชจ๋ธ๋“ค์ด ํ•˜๋Š” ๊ฒƒ์„ ๊ฒฐ๊ตญ transformer์—์„œ self-attention, multi-head๋กœ ๋‹ค ํ•˜๊ณ , ์‹ค์ œ๋กœ ์„ฑ๋Šฅ๋„ ๋” ์ข‹์•˜๋‹ค. ๋Œ€์‹ ์— ๋ ˆ์ดํ„ด์‹œ๊ฐ€ ๋ฌธ์ œ^^
    • GPU์„œ๋น™ + ๋ชจ๋ธ ์ตœ์ ํ™”๋กœ ํ•ด๊ฒฐ
  • Sequence Features
    • ๋ชจ๋ธ ์šฉ๋Ÿ‰์ด ๋” ์ปค์ง€๋‹ˆ๊นŒ sequence ๋ฅผ ๋ฐ”๋กœ ๋ชจ๋ธ์— ๋„ฃ์„ ์ˆ˜ ์žˆ๊ณ , ํšจ๊ณผ๊ฐ€ ์ข‹์•˜๋‹ค๊ณ  ํ•œ๋‹ค
  • Position Bias
    • Position๋„ ํŠธ๋ ˆ์ด๋‹์— ํฌํ•จํ•œ ๋‹ค์Œ์— ์„œ๋น™ํƒ€์ž„์—๋Š” ์ด๊ฑธ ๋นผ๋Š” ์‹์œผ๋กœ ํ•ด๊ฒฐ

๋Š๋‚€์ 

  • Amazon์—์„œ ๋‚˜์˜จ DCAF-BERT ๋ผ๋Š” ํŽ˜์ดํผ๋„ ๋ดค์—ˆ๋Š”๋ฐ ์ ์  CTR ๋ชจ๋ธ, Engagement ๋ชจ๋ธ ์ด๋Ÿฐ๊ฑฐ๋„ transformer๋กœ ๊ฐ€๋Š” ๊ฒƒ ๊ฐ™๋‹ค. CTR ๋ชจ๋ธ์— ๋Œ€ํ•ด ๊ณต๋ถ€๋ฅผ ํ•˜๋‹ค๋ณด๋ฉด DCN, DIEN, CAN ๋“ฑ๋“ฑ ๋„คํŠธ์›Œํฌ ๊ตฌ์กฐ๊ฐ€ ์—„์ฒญ ๋งŽ์ด ๋‚˜์˜ค๋Š”๋ฐ ์„œ๋น™ ์ธํ”„๋ผ๊ฐ€ ์ ์  ๋ฐœ๋‹ฌํ•˜๋ฉด ์ด๋Ÿฐ๊ฒŒ ๋œ ์ค‘์š”ํ•ด์ง€์ง€ ์•Š์„๊นŒ..?
  • ์˜จ๋ผ์ธ ์„œ๋น™์€ ๋ ˆ์ดํ„ด์‹œ์— ์˜ˆ๋ฏผํ•˜๋‹ˆ๊นŒ ์–ด๋Š ์„ ๊นŒ์ง€๋Š” PinSAGE, PinnerFormer ์ฒ˜๋Ÿผ representation์— ์ž˜ ํˆฌ์žํ•˜๋Š” ๊ฒƒ์ด ์„œ๋น™ ์ธํ”„๋ผ๋ฅผ ์—…๊ทธ๋ ˆ์ด๋“œ๋ฅผ ํ•˜์ง€ ์•Š์œผ๋ฉด์„œ๋„ ๋”ฅ๋Ÿฌ๋‹์˜ ๋•์„ ๋ณผ ์ˆ˜ ์žˆ๋Š” ๋ฐฉ๋ฒ• ๊ฐ™๋‹ค.

Published Apr 23, 2022

If I keep marking the dots, someday they will ๐Ÿ”—๐Ÿ”—