中文标题#
一种多方面评估框架，用于评估由大型语言模型生成的合成数据
英文标题#
A Multi-Faceted Evaluation Framework for Assessing Synthetic Data Generated by Large Language Models
中文摘要#
生成式人工智能和大型语言模型（LLMs）的快速发展为生成合成数据开辟了新的途径，尤其是在结构化表格格式领域，例如产品评论。 尽管有潜在的好处，但关于隐私泄露的担忧已经出现，特别是在训练数据集中使用个人信息时。 此外，缺乏一个全面的评估框架，该框架能够定量衡量生成的合成数据的质量及其对下游任务的实用性。 为了解决这一空白，我们引入了 SynEval，这是一个开源的评估框架，旨在通过一系列多样的评估指标来评估合成生成的表格数据的真实性、实用性和隐私保护。 我们通过将 SynEval 应用于由三种最先进的 LLMs：ChatGPT、Claude 和 Llama 生成的合成产品评论数据来验证我们提出的框架的有效性。 我们的实验结果揭示了在合成数据生成背景下各种评估指标之间的权衡。 此外，SynEval 是研究人员和从业者处理合成表格数据的重要工具，使他们能够谨慎地确定生成的数据是否适合其特定应用，特别强调维护用户隐私。
英文摘要#
The rapid advancements in generative AI and large language models (LLMs) have opened up new avenues for producing synthetic data, particularly in the realm of structured tabular formats, such as product reviews. Despite the potential benefits, concerns regarding privacy leakage have surfaced, especially when personal information is utilized in the training datasets. In addition, there is an absence of a comprehensive evaluation framework capable of quantitatively measuring the quality of the generated synthetic data and their utility for downstream tasks. In response to this gap, we introduce SynEval, an open-source evaluation framework designed to assess the fidelity, utility, and privacy preservation of synthetically generated tabular data via a suite of diverse evaluation metrics. We validate the efficacy of our proposed framework - SynEval - by applying it to synthetic product review data generated by three state-of-the-art LLMs: ChatGPT, Claude, and Llama. Our experimental findings illuminate the trade-offs between various evaluation metrics in the context of synthetic data generation. Furthermore, SynEval stands as a critical instrument for researchers and practitioners engaged with synthetic tabular data,, empowering them to judiciously determine the suitability of the generated data for their specific applications, with an emphasis on upholding user privacy.
PDF 获取#
抖音扫码查看更多精彩内容