Generator Kisah Daerah Berbasis Bahasa Jawa dengan Finetuned GPT-2

Penulis

  • Kayla Queenazima Santoso Fakultas Teknik, Universitas Gadjah Mada
  • Karunia Perjuangan Mustadl’afin Fakultas Teknik, Universitas Gadjah Mada
  • Lutfi Andriyanto Fakultas Teknik, Universitas Gadjah Mada
  • Igi Ardiyanto Fakultas Teknik, Universitas Gadjah Mada

Kata Kunci:

Text Generator Berbahasa Jawa, GPT-2, Large Language Model, Low-Resource Language Model

Abstrak

Pengembangan text generator pada Bahasa Jawa bertujuan menciptakan generator kisah daerah yang memanfaatkan dua skema dataset, yaitu data hasil translasi dan data teks/dokumen digital berbahasa Jawa dengan variasi dua model pre-trained GPT-2. Pendekatan ini dimaksudkan untuk mengatasi masalah kualitas dan kehilangan unsur karakter kultural serta linguistik dalam proses pengumpulan dataset. Dengan demikian, pengembangan ini diharapkan dapat meningkatkan ekonomi dan pelestarian budaya Indonesia melalui kreativitas storytelling berbahasa jawa. Metode yang digunakan ialah fine-tuning GPT-2 pre-trained. Dari pengembangan metode tersebut, didapatkan skema model finetuned GPT-2 Medium Berbahasa Indonesia dengan skema dataset II serta scheduler inverse square root dengan perplexity 65,70.

Referensi

이제용, J., & Yoo, K. W. (2018). The structural relationship between selection attributes of traditional culture festival storytelling, brand equity, brand attitude, and brand loyalty. The Journal of the Korea Contents Association, 18, 647-659.

Kementerian Pendidikan, Kebudayaan, Riset, dan Teknologi, "Badan Bahasa petakan 652 bahasa daerah di Indonesia," Kementerian Pendidikan, Kebudayaan, Riset, dan Teknologi, [Online]. Available: https://www.kemdikbud.go.id/main/blog/2018/07/badan-bahasa-petakan-652-bahasa-daerah-di-indonesia. [Accessed: Jul. 1, 2023].

D. M. Eberhard, G. F. Simons, and C. D. Fennig (eds.), "Ethnologue: Languages of the World," 26th ed. Dallas, Texas: SIL International, 2023. [Online]. Available: http://www.ethnologue.com.

R. Wijaya and B. Pudjoatmodjo, "An overview and implementation of extraction-transformation-loading (ETL) process in data warehouse (Case study: Department of agriculture)," 2015 3rd International Conference on Information and Communication Technology (ICoICT), Nusa Dua, Bali, Indonesia, 2015, pp. 70-74, doi: 10.1109/ICoICT.2015.7231399.

Radford, A., Narasimhan, K., Salimans, T., dan Sutskever, I. 2018. Improving Language Understanding by Generative Pre-Training. OpenAI.

Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł,. dan Polosukhin, I. 2017. Attention is all you need. In Advances in neural information processing systems. pp.5998-6008.

Z. Chi, H. Huang, L. Liu, Y. Bai, X. Gao and X. -L. Mao, "Can Pretrained English Language Models Benefit Non-English NLP Systems in Low-Resource Scenarios?," in IEEE/ACM Transactions on Audio, Speech, and Language Processing, doi: 10.1109/TASLP.2023.3267618.

W. Wongso, D. S. Setiawan and D. Suhartono, "Causal and Masked Language Modeling of Javanese Language using Transformer-based Architectures," 2021 International Conference on Advanced Computer Science and Information Systems (ICACSIS), Depok, Indonesia, 2021, pp. 1-7, doi: 10.1109/ICACSIS53237.2021.9631331.

Akmal et al., “Indonesian-NLP/GPT2-medium-Indonesian · hugging face,” indonesian-nlp/gpt2-medium-indonesian · Hugging Face, https://huggingface.co/indonesian-nlp/gpt2-medium-indonesian (accessed Jul. 1, 2023).

I. Zafar and G. Tzanidou, Hands-on Convolutional Neural Networks with Tensorflow: Solve Computer Vision Problems with Modeling in Tensorflow and Python. Birmingham ; Mumbai: Packt, 2018.

Perplexity of fixed-length models, https://huggingface.co/docs/transformers/perplexity (accessed Jun. 30, 2023).

Unduhan

Diterbitkan

30-04-2024