中文标题#
一種多方面評估框架，用於評估由大型語言模型生成的合成數據
英文标题#
A Multi-Faceted Evaluation Framework for Assessing Synthetic Data Generated by Large Language Models
中文摘要#
生成式人工智慧和大型語言模型（LLMs）的快速發展為生成合成數據開闢了新的途徑，尤其是在結構化表格格式領域，例如產品評論。 儘管有潛在的好處，但關於隱私洩露的擔憂已經出現，特別是在訓練數據集中使用個人信息時。 此外，缺乏一個全面的評估框架，該框架能夠定量衡量生成的合成數據的質量及其對下游任務的實用性。 為了解決這一空白，我們引入了 SynEval，這是一個開源的評估框架，旨在通過一系列多樣的評估指標來評估合成生成的表格數據的真實性、實用性和隱私保護。 我們通過將 SynEval 應用於由三種最先進的 LLMs：ChatGPT、Claude 和 Llama 生成的合成產品評論數據來驗證我們提出的框架的有效性。 我們的實驗結果揭示了在合成數據生成背景下各種評估指標之間的權衡。 此外，SynEval 是研究人員和從業者處理合成表格數據的重要工具，使他們能夠謹慎地確定生成的數據是否適合其特定應用，特別強調維護用戶隱私。
英文摘要#
The rapid advancements in generative AI and large language models (LLMs) have opened up new avenues for producing synthetic data, particularly in the realm of structured tabular formats, such as product reviews. Despite the potential benefits, concerns regarding privacy leakage have surfaced, especially when personal information is utilized in the training datasets. In addition, there is an absence of a comprehensive evaluation framework capable of quantitatively measuring the quality of the generated synthetic data and their utility for downstream tasks. In response to this gap, we introduce SynEval, an open-source evaluation framework designed to assess the fidelity, utility, and privacy preservation of synthetically generated tabular data via a suite of diverse evaluation metrics. We validate the efficacy of our proposed framework - SynEval - by applying it to synthetic product review data generated by three state-of-the-art LLMs: ChatGPT, Claude, and Llama. Our experimental findings illuminate the trade-offs between various evaluation metrics in the context of synthetic data generation. Furthermore, SynEval stands as a critical instrument for researchers and practitioners engaged with synthetic tabular data, empowering them to judiciously determine the suitability of the generated data for their specific applications, with an emphasis on upholding user privacy.
PDF 获取#
抖音掃碼查看更多精彩內容