All Articles

Zero-shot Neural Passage Retrieval via Domain-targeted Synthetic Question Generation

https://www.aclweb.org/anthology/2021.eacl-main.92/

์งง์€ ๋ฒ„์ „

์–ด๋–ค ๋„๋ฉ”์ธ์— ๋Œ€ํ•ด์„œ ๋ฌธ์„œ๋งŒ ๋ฉ๊ทธ๋Ÿฌ๋‹ˆ ์žˆ์„ ๋•Œ, ์ผ๋ฐ˜์ ์ธ ๋„๋ฉ”์ธ์—์„œ ํ›ˆ๋ จ๋œ ๋ฌธ์„œ -> ์งˆ๋ฌธ์„ ๋งŒ๋“œ๋Š” ๋ชจ๋ธ์„ ์ด์šฉํ•ด์„œ syntheticํ•œ <๋ฌธ์„œ, ์งˆ๋ฌธ>์˜ pair๋ฅผ ๋งŒ๋“ค ์ˆ˜ ์žˆ๋‹ค. ์ด๊ฑธ ๊ฐ€์ง€๊ณ  siamese BERT๋ฅผ ํ›ˆ๋ จํ•˜๋ฉด ์ข‹์€ ํ’ˆ์งˆ์˜ ์ž„๋ฒ ๋”ฉ์„ ์–ป์„ ์ˆ˜ ์žˆ๋‹ค. supervised ํŠธ๋ ˆ์ด๋‹์„ ์œ„ํ•œ ๋ฐ์ดํ„ฐ๊ฐ€ ์กด์žฌํ•˜๋Š” ๋ฐ์ดํ„ฐ์…‹์— ๋Œ€ํ•ด์„œ ์œ„ ๊ณผ์ •์„ ๊ฑฐ์นœ ์ž„๋ฒ ๋”ฉ๊ณผ BM25๋ฅผ concatํ•œ ๋ฒกํ„ฐ๋ฅผ ๊ฐ€์ง€๊ณ  nearest neighbor ๊ฒ€์ƒ‰๋งŒ ํ•ด๋„, BM25 + supervised ํŠธ๋ ˆ์ด๋‹์„ ๊ฑฐ์นœ ๋ชจ๋ธ์„ ํ•ฉ๋งŒํผ ์„ฑ๋Šฅ์ด ๋‚˜์˜จ๋‹ค.

์ž˜ ๋ชฐ๋ž๋˜ ๊ฒƒ

๋กœ์Šค ํ•จ์ˆ˜๋Š” Batch Loss๋ผ๊ณ , batchํ•˜๋‚˜์— Q, P+, (P-, P-, P-..) ์‹์œผ๋กœ ๊ด€๋ จ์žˆ๋Š” Passage ํ•˜๋‚˜, ๋‚˜๋จธ์ง€๋Š” ๊ด€๋ จ์—†๋Š” Passage๋กœ ์ฑ„์šฐ๊ณ  loss๋ฅผ ๊ณ„์‚ฐํ•˜์…จ๋‹ค๊ณ  ํ•˜์…จ๋‹ค. Q๋ž‘ P+๋Š” ๋ฐ์ดํ„ฐ์…‹์— ์žˆ๋‹ค๊ณ  ์น˜๋ฉด P-๋Š” ์–ด๋””์„œ ๊ฐ€์ ธ์™€์•ผํ•˜๋‚˜? in-batch negative ๋ฅผ ์‚ฌ์šฉํ•œ๋‹ค๊ณ  ํ•˜์…จ๋‹ค. ๋ฉ์ฒญํ•˜๊ฒŒ in-batch negatvie๋ฅผ ๋ณผ ์ƒ๊ฐ์€ ์•ˆํ•˜๊ณ  ๋‹ค๋ฅธ๊ฑฐ ๋ถ€ํ„ฐ ์ฐพ์•˜๋‹ค -_-

  1. ๋ผ๋ฒจ ๊ฐ™์€๊ฒŒ ๋”ฐ๋กœ ์—†์œผ๋ฉด ๊ทธ๋ƒฅ ๋žœ๋ค์ƒ˜ํ”Œ๋ง https://github.com/awslabs/neural-retrieval/blob/main/data/data.py#L156
  2. ๋ผ๋ฒจ์ด ์žˆ์œผ๋ฉด ๋‹ค๋ฅธ ๋ผ๋ฒจ์—์„œ extractํ•˜๊ธฐ https://www.kaggle.com/xhlulu/shopee-generate-data-for-triplet-loss

in-batch negative๋ผ๊ณ  ๋ง๋งŒํ•˜๊ณ  ์™œ ๋„˜์–ด๊ฐ€๋‚˜ ํ–ˆ๋”๋‹ˆ ์ด๊ฑธ ์ฒ˜์Œ ์†Œ๊ฐœํ•œ ๋…ผ๋ฌธ์— ์„ค๋ช…์ด ์ž˜ ๋˜์žˆ์–ด์„œ ์•ˆ ํ•œ ๊ฒƒ์ด๋‹ค. https://github.com/facebookresearch/DPR/blob/master/train_dense_encoder.py#L646 ์—ฌ๊ธฐ๋ถ€ํ„ฐ ๋ณด๋ฉด ๋œ๋‹ค (์‚ฌ์‹ค ๋ณดํ†ต ์‚ฌ๋žŒ๋“ค์€ DPR์„ ๋ณด๊ณ  ์ด ๋…ผ๋ฌธ์„ ๋ดค์„ํ…Œ๋‹ˆ ๋‹น์—ฐํ•œ ๊ฒƒ)

ํ•œ๊ตญ์–ด์—์„œ?

D -> Q๊ฐ€ ์ž˜๋œ๋‹ค๋Š” ์ „์ œ ํ•˜์—์„œ ์ด๊ฒŒ ๊ฐ€๋Šฅํ•œ๊ฑฐ๊ณ , ๊ทธ๋Ÿด๋ ค๋ฉด ์–ด๋Š์ •๋„ ํ€„๋ฆฌํ‹ฐ์žˆ๋Š” (Q, D) ํŽ˜์–ด๊ฐ€ ๋งŽ์•„์•ผ ํ•˜๋Š”๋ฐ ์–ผ๋งˆ๋‚˜ ๋งŽ์ด ์Œ“์—ฌ ์žˆ๋Š”์ง€ ๋ชจ๋ฅด๊ฒ ๋‹ค. KLUE ๊ฐ™์€๊ฒŒ ์—…๊ณ„์— ์ •๋ง ํฐ ๊ธฐ์—ฌ ๊ฐ™๋‹ค.

Published Jun 12, 2021

If I keep marking the dots, someday they will ๐Ÿ”—๐Ÿ”—