Haystack 文本提取器深度指南：LLMDocumentContentExtractor、LLMMetadataExtractor 与 RegexTextExtractor

Haystack 文本提取器深度指南LLMDocumentContentExtractor、LLMMetadataExtractor 与 RegexTextExtractor【免费下载链接】haystackOpen-source AI orchestration framework for building context-engineered, production-ready LLM applications. Design modular pipelines and agent workflows with explicit control over retrieval, routing, memory, and generation. Built for scalable agents, RAG, multimodal applications, semantic search, and conversational systems.项目地址: https://gitcode.com/GitHub_Trending/ha/haystackHaystack 是开源、面向生产环境的 LLM 应用编排框架其 extractors 组件族 专门负责从文档中抽取结构化信息。本文以 extractors_api.md 为骨架结合 源码 与 测试用例系统讲解LLMDocumentContentExtractor图像文档内容抽取、LLMMetadataExtractorLLM 元数据抽取与RegexTextExtractor正则文本抽取三大组件的设计原理、参数语义、运行流程与实战用法读完即可在索引、RAG 与 Agent 流程中落地使用。一、组件总览三种抽取器各司其职haystack/components/extractors包由三个模块组成模块入口在 extractors/init.py并通过 pydoc/extractors_api.yml 生成 API 文档组件模块路径输入输出能力LLMDocumentContentExtractorimage/llm_document_content_extractor.py图像/PDF 文件路径存于文档 meta将图像内容抽取为文本content并可附带元数据LLMMetadataExtractorllm_metadata_extractor.py文本 Documents为每个文档生成结构化元数据如 NER 实体RegexTextExtractorregex_text_extractor.py字符串或ChatMessage列表用正则捕获组提取指定文本片段从源码结构看三者都通过component装饰器注册为标准 Haystack 组件均提供run/run_async或其中一种与to_dict/from_dict序列化能力可以无缝嵌入 Pipeline。其中LLMDocumentContentExtractor与LLMMetadataExtractor属于LLM 驱动型抽取器RegexTextExtractor则是零依赖的规则型抽取器。二、LLMDocumentContentExtractor用视觉 LLM 抽取图像文档2.1 核心思想与工作流LLMDocumentContentExtractor面向图像型文档扫描件、截图、PDF 页面渲染图借助支持视觉输入的 ChatGenerator 完成内容抽取。其内部流程如下每个文档一次 prompt、一次 LLM 调用通过DocumentToImageContent见 converters/image/document_to_image.py将文档转换为ImageContent——它支持直接图像文件也支持从 PDF 按page_number渲染指定页将 prompt 与图像组装成多模态ChatMessageTextContentImageContent见 llm_document_content_extractor.py发给 ChatGenerator按约定的响应格式解析结果回填文档的content与meta。组件内建默认 prompt 模板DEFAULT_PROMPT_TEMPLATE源码 L33-L61要求模型精确抽取、按 markdown 排版、保持阅读顺序对图表用[img-caption][/img-caption]添加说明、表格用[table-caption][/table-caption]加注表单勾选以 markdown 还原最终返回含document_content键的 JSON 对象。2.2 响应处理规则重点组件对 LLM 返回文本采用统一解析逻辑_process_response源码 L256-L274纯字符串非 JSON 或非 JSON 对象整个字符串直接写入文档content仅含document_content键的 JSON 对象该键的值写入content含多个键的 JSON 对象document_content的值写入content其余键值对合并进文档 meta例如 author、date、title 等合法 JSON 但不是对象数组或原始值报告错误文档进入failed_documents。因此为了让模型稳定输出结构化结果推荐在generation_kwargs中配置response_format{type: json_object}或更严格的json_schema。2.3 参数说明__init__签名源码 L136-L177__init__( *, chat_generator: ChatGenerator, prompt: str DEFAULT_PROMPT_TEMPLATE, file_path_meta_field: str file_path, root_path: str | None None, detail: Literal[auto, high, low] | None None, size: tuple[int, int] | None None, raise_on_failure: bool False, max_workers: int 3 ) - None参数类型默认值说明chat_generatorChatGenerator必填支持视觉输入的 ChatGenerator可选配置 JSON 输出promptstr内建模板抽取指令严禁包含 Jinja 变量构造时用沙箱环境解析并校验见 L245-L254file_path_meta_fieldstrfile_path文档 meta 中保存图像/PDF 路径的字段名root_pathstr \| NoneNone文档文件所在根目录设置后路径按相对根目录解析并强制限定在该目录内用于防御路径穿越detailauto\|high\|lowNone图像细节级别仅 OpenAI 支持sizetuple[int,int]None等比缩放到 (宽, 高) 范围内降低传输与处理开销raise_on_failureboolFalseTrue 时 LLM 异常直接抛出False 时失败文档进入failed_documentsmax_workersint3并行 LLM 调用的最大线程数安全提示务必阅读该组件会读取file_path_meta_field指向的宿主机文件。若文档 meta 可能受不可信输入影响必须设置root_path为专用数据目录使绝对路径或../等路径穿越载荷被拒绝而非读取。这一约束同样内置于DocumentToImageContentdocument_to_image.py L81-L86。2.4 完整用法示例以下示例取自 API 文档使用OpenAIChatGenerator并配置json_schema强制输出document_content与可选元数据字段from haystack import Document from haystack.components.generators.chat import OpenAIChatGenerator from haystack.components.extractors.image import LLMDocumentContentExtractor prompt Extract the content from the provided image. Format everything as markdown. Return only the extracted content as a JSON object with the key document_content. No markdown, no code fence, only raw JSON. Extract metadata about the image like source of the image, date of creation, etc. if you can. Return this metadata as additional key-value pairs in the same JSON object. chat_generator OpenAIChatGenerator( generation_kwargs{ response_format: { type: json_schema, json_schema: { name: entity_extraction, schema: { type: object, properties: { document_content: {type: string}, author: {type: string}, date: {type: string}, document_type: {type: string}, title: {type: string}, }, additionalProperties: False, }, }, } } ) extractor LLMDocumentContentExtractor( chat_generatorchat_generator, file_path_meta_fieldfile_path, raise_on_failureFalse, ) documents [ Document(content, meta{file_path: test/test_files/images/image_metadata.png}), Document(content, meta{file_path: test/test_files/images/apple.jpg, page_number: 1}), ] result extractor.run(documentsdocuments) updated_documents result[documents]注意page_number用于指示 PDF 文档要渲染的页码若传入的是图片文件则无需该字段。失败文档会进入result[failed_documents]其 meta 中带有content_extraction_error键测试见 test_llm_document_content_extractor.py 中test_run_with_llm_failure_raise_on_failure_false等用例。2.5 生命周期方法与异步支持warm_up()/warm_up_async()预热底层 ChatGenerator异步版本优先调用warm_up_async否则回退同步warm_upclose()/close_async()释放底层生成器资源run_async()异步版本LLM 调用并发执行但受max_workers信号量约束若生成器仅实现同步run会在线程中执行以避免阻塞事件循环图片文件读取与 PDF 渲染属于阻塞操作会在线程中执行源码 L407-L451。测试test_run_async_falls_back_to_sync_run、test_run_async_converts_images_off_the_event_loop、test_run_async_respects_max_workers对此均有覆盖。三、LLMMetadataExtractorLLM 驱动的结构化元数据抽取3.1 设计原理LLMMetadataExtractorllm_metadata_extractor.py接收 Documents 列表与一个 prompt对每个文档分别运行一次 LLM将抽取结果合并进文档 meta。典型场景是 NER 实体识别、文档标签、摘要字段等。其 prompt 约束是关键设计prompt 中必须有且仅有一个变量document指向列表中的单个文档例如{{ document.content }}组件在__init__中通过SandboxedEnvironment().parse(prompt)解析模板变量若变量集合不等于[document]会直接抛出ValueError源码 L187-L194组件内部用PromptBuilder(prompt, required_variablesvariables)渲染 prompt再包装为ChatMessage.from_user(...)调用生成器源码 L285-L312。3.2 参数说明__init__( prompt: str, chat_generator: ChatGenerator, expected_keys: list[str] | None None, page_range: list[str | int] | None None, raise_on_failure: bool False, max_workers: int 3, ) - None参数类型默认值说明promptstr必填模板 prompt必须恰好含一个变量documentchat_generatorChatGenerator必填LLM 实例建议配置generation_kwargs{response_format: {type: json_object}}强制 JSON 输出expected_keyslist[str]None期望 LLM JSON 输出中包含的键解析时校验缺失键见_extract_metadataL271-L283page_rangelist[str\|int]None按页抽取范围可被run方法覆盖raise_on_failureboolFalseLLM 执行或 JSON 校验失败时是否抛出异常max_workersint3线程池最大并发数run_async中用作并发上限3.3 page_range按页抽取page_range支持单页与可打印范围字符串由expand_page_range展开haystack/utils/misc.py[1, 3]→ 抽取第 1、3 页[1-3, 5, 8, 10-12]→ 展开为 1,2,3,5,8,10,11,12传入整数时若含-会被拒绝要求范围必须是start-end字符串空结果会抛出ValueError。实现上组件内部创建DocumentSplitter(split_bypage, split_length1)源码 L198在_prepare_prompts中按展开后的页码拼接目标页文本再填充进 prompt不传page_range时对整篇文档抽取。run与run_async均支持运行期覆盖page_range。3.4 失败处理与重跑机制这是该组件最具实战价值的设计失败文档进入failed_documentsmeta 中写入metadata_extraction_error错误信息与metadata_extraction_responseLLM 原始回复供后续重试参考成功重跑后组件会清除之前遗留的metadata_extraction_error/metadata_extraction_response键源码 L378-L382可将metadata_extraction_response与metadata_extraction_error重新注入 prompt用另一个抽取器对失败文档二次抽取。3.5 NER 实战示例以下为 API 文档的完整 NER 示例使用OpenAIChatGeneratorjson_schema强制输出entities数组from haystack import Document from haystack.components.extractors.llm_metadata_extractor import LLMMetadataExtractor from haystack.components.generators.chat import OpenAIChatGenerator NER_PROMPT -Goal- Given text and a list of entity types, identify all entities of those types from the text. -Steps- 1. Identify all entities. For each identified entity, extract the following information: - entity: Name of the entity - entity_type: One of the following types: [organization, product, service, industry] Format each entity as a JSON like: {entity: entity_name, entity_type: entity_type} 2. Return output in a single list with all the entities identified in steps 1. -Examples- ###################### Example 1: entity_types: [organization, person, partnership, financial metric, product, service, industry, investment strategy, market trend] text: Another area of strength is our co-brand issuance. Visa is the primary network partner for eight of the top 10 co-brand partnerships in the US today and we are pleased that Visa has finalized a multi-year extension of our successful credit co-branded partnership with Alaska Airlines, a portfolio that benefits from a loyal customer base and high cross-border usage. ... output: {entities: [{entity: Visa, entity_type: company}, {entity: Alaska Airlines, entity_type: company}, ...]} ############################# -Real Data- ###################### entity_types: [company, organization, person, country, product, service] text: {{ document.content }} ###################### output: docs [ Document(contentdeepset was founded in 2018 in Berlin, and is known for its Haystack framework), Document(contentHugging Face is a company that was founded in New York, USA and is known for its Transformers library) ] chat_generator OpenAIChatGenerator( generation_kwargs{ max_completion_tokens: 500, temperature: 0.0, seed: 0, response_format: { type: json_schema, json_schema: { name: entity_extraction, schema: { type: object, properties: { entities: { type: array, items: { type: object, properties: { entity: {type: string}, entity_type: {type: string} }, required: [entity, entity_type], additionalProperties: False } } }, required: [entities], additionalProperties: False } } }, }, max_retries1, timeout60.0, ) extractor LLMMetadataExtractor( promptNER_PROMPT, chat_generatorchat_generator, expected_keys[entities], raise_on_failureFalse, ) result extractor.run(documentsdocs)输出示意来自 API 文档{documents: [ Document(id.., content: deepset was founded in 2018 in Berlin, ..., meta: {entities: [{entity: deepset, entity_type: company}, {entity: Berlin, entity_type: city}, {entity: Haystack, entity_type: product}]}), Document(id.., content: Hugging Face is a company ..., meta: {entities: [{entity: Hugging Face, entity_type: company}, {entity: New York, entity_type: city}, {entity: USA, entity_type: country}, {entity: Transformers, entity_type: product}]}) ], failed_documents: []}要点提示expected_keys[entities]会在_extract_metadata中校验输出 JSON 是否含该键推荐temperature0.0、seed0提升确定性json_schema比json_object约束更严格能显著降低解析失败率若 LLM 输出非合法 JSON 或缺失键且raise_on_failureFalse文档会进入failed_documents可通过metadata_extraction_response人工检查或重试。四、RegexTextExtractor零依赖的正则文本抽取4.1 功能与用法RegexTextExtractorregex_text_extractor.py是三者中最轻量的组件给定一个含捕获组的正则从字符串或ChatMessage列表中提取匹配文本。适用于从 LLM 输出或结构化文本中提取 URL、ID、命令等确定模式无需调用任何模型。from haystack.components.extractors import RegexTextExtractor from haystack.dataclasses import ChatMessage # 传入字符串 parser RegexTextExtractor(regex_patternissue url(.)) result parser.run(text_or_messagesissue urlgithub.com/hahahahahahahah/issue) # result: {captured_text: github.com/hahahaha} # 传入 ChatMessages仅处理最后一条消息 messages [ChatMessage.from_user(issue urlgithub.com/hahahahahahahah/issue)] result parser.run(text_or_messagesmessages) # result: {captured_text: github.com/hahahaha}4.2 行为细节从源码与测试印证__init__(regex_pattern: str)构造时用re.compile(...).groups检查捕获组数量若无捕获组会打印警告整个匹配将被返回源码 L52-L59此时返回match.group(0)即完整匹配run(text_or_messages: str | list[ChatMessage]) - dict[str, str]输入为字符串直接re.search提取输入为ChatMessage列表仅处理最后一条消息_process_last_message若末元素不是ChatMessage实例抛出TypeError空列表返回{captured_text: }有捕获组时返回第 1 个捕获组match.group(1)无匹配返回{captured_text: }序列化to_dict/from_dict仅保留regex_patternfrom_dict会兼容清理旧版本遗留的return_empty_on_no_match参数源码 L80-L84保证老 Pipeline 反序列化不中断。4.3 与 LLM 抽取器的分工维度RegexTextExtractorLLMMetadataExtractorLLMDocumentContentExtractor依赖无需 LLMChatGenerator需视觉 LLMChatGenerator适用输入字符串 / ChatMessage文本 Documents图像 / PDF Documents输出captured_text合并进meta写入contentmeta失败处理返回空字符串failed_documents 重跑机制failed_documentscontent_extraction_error异步仅同步runrun/run_asyncrun/run_async实际 Pipeline 中常见组合先用RegexTextExtractor做确定性的轻量抽取如抓取工具调用返回中的 URL再对需要语义理解的部分交给LLMMetadataExtractor或LLMDocumentContentExtractor两类组件共享ChatGenerator接口便于替换后端模型。五、序列化与 Pipeline 集成三个组件都实现了标准的to_dict/from_dictLLMDocumentContentExtractor.to_dict会序列化chat_generator通过component_to_dict、prompt、file_path_meta_field、root_path、detail、size、raise_on_failure、max_workersfrom_dict通过deserialize_chatgenerator_inplace原地还原生成器源码 L230-L243LLMMetadataExtractor额外序列化expected_keys与展开后的page_range源码 L239-L255RegexTextExtractor仅序列化regex_pattern。这意味着三者都可以直接嵌入 Haystack Pipeline 用 YAML 声明type: haystack.components.extractors.llm_metadata_extractor.LLMMetadataExtractor等与 DocumentSplitter 组合做按页/按块批量抽取配合DocumentWriter将更新后的文档含新 meta写入 DocumentStore。六、总结本文以 extractors_api.md 为主线完整覆盖了 Haystack 三大文本抽取组件的设计、参数、运行流程与实战用法LLMDocumentContentExtractor视觉 LLM DocumentToImageContent完成图像/PDF 文档内容抽取支持内容回填与元数据合并务必关注root_path路径安全LLMMetadataExtractor单变量 prompt 约束 JSON 校验 page_range按页抽取 失败重跑机制是 RAG 元数据增强与 NER 的通用方案RegexTextExtractor轻量确定性抽取与 LLM 抽取器形成互补。三者共享 Haystack 的组件协议生命周期、序列化、同步/异步可无缝组合进生产级 Pipeline。深入阅读可继续查看源码实现 haystack/components/extractors 与测试用例 test/components/extractors。【免费下载链接】haystackOpen-source AI orchestration framework for building context-engineered, production-ready LLM applications. Design modular pipelines and agent workflows with explicit control over retrieval, routing, memory, and generation. Built for scalable agents, RAG, multimodal applications, semantic search, and conversational systems.项目地址: https://gitcode.com/GitHub_Trending/ha/haystack创作声明：本文部分内容由AI辅助生成（AIGC），仅供参考