← Back to Development

Unified Document Parser

Architecture & Design Document | Version 1.0 | Created: April 2026

System Overview

The Unified Document Parser is a centralized system designed to ingest, parse, deduplicate, and correlate data from multiple document sources. The system extracts actionable instructions and formats data for database storage and downstream processing via the Neo/Recall module.

Data Flow Architecture

INPUT SOURCES
Microsoft 365 Mailbox | PDF Documents
↓
PARSING LAYER
Email Parser | PDF Parser
Unified helper/parser system
↓
CORRELATION ENGINE
Deduplication | Cross-referencing
Extract Solutions/Instructions
(e.g., "bring to local distribution center")
↓
FORMATTING LAYER
DB Schema Transformation
Data Normalization
↓
OUTPUT
Neo/Recall Module/API TBD

System Components

📧 Email Parser Planned

  • Microsoft 365 integration
  • Extract message content, attachments, metadata
  • Handle various email formats (HTML, plain text)
  • Extract actionable items and instructions

📄 PDF Parser Planned

  • Extract text, tables, and structured data
  • Handle scanned documents (OCR)
  • Preserve document structure and formatting
  • Extract embedded instructions and metadata

🔧 Unified Helper System Planned

  • Common parsing utilities and interfaces
  • Standardized data extraction methods
  • Error handling and logging
  • Plugin architecture for extensibility

🔄 Deduplication Engine Planned

  • Content-based hashing
  • Fuzzy matching algorithms
  • Cross-source duplicate detection
  • Merge duplicate records intelligently

🔗 Correlation Engine Planned

  • Link related documents across sources
  • Identify reference patterns
  • Extract and standardize instructions
  • Build document relationship graphs

💾 DB Schema Formatter Planned

  • Transform parsed data to target schema
  • Data validation and sanitization
  • Handle missing or malformed fields
  • Generate database-ready payloads

🚀 Neo/Recall Integration TBD

  • API client for Neo/Recall module
  • Batch processing support
  • Retry logic and error recovery
  • Integration method to be determined

Key Features

Core Capabilities

  • Multi-source ingestion: Seamlessly handle both email and PDF document sources
  • Intelligent deduplication: Prevent duplicate processing across sources
  • Cross-document correlation: Link related information from different sources
  • Instruction extraction: Parse and standardize actionable items (e.g., delivery instructions)
  • Schema-agnostic design: Flexible formatting layer adapts to target database requirements
  • Modular architecture: Easy to extend with new parsers or data sources

Technical Considerations

Parser Design Patterns

  • Strategy Pattern: Unified interface with source-specific implementations
  • Factory Pattern: Dynamic parser instantiation based on document type
  • Chain of Responsibility: Multi-stage processing pipeline for complex documents

Deduplication Strategy

  • Content Hashing: SHA-256 hashes for exact duplicate detection
  • Fuzzy Matching: Levenshtein distance or semantic similarity for near-duplicates
  • Metadata Comparison: Timestamps, senders, subjects as secondary signals
  • Bloom Filters: Fast preliminary duplicate checks for high-volume scenarios

Correlation Approaches

  • Reference ID Extraction: Parse tracking numbers, order IDs, case numbers
  • Entity Recognition: NER for people, organizations, locations, dates
  • Temporal Correlation: Link documents based on time windows
  • Instruction Standardization: Map varied phrasings to canonical actions

Data Flow Considerations

  • Async Processing: Queue-based architecture for handling high volumes
  • Error Recovery: Dead letter queues and retry mechanisms
  • Observability: Comprehensive logging, metrics, and tracing
  • Schema Versioning: Handle DB schema evolution gracefully

Example Use Cases

Scenario 1: Distribution Center Instructions

An email arrives with: "Please bring package #12345 to local distribution center by EOD"
A PDF invoice contains: "PKG-12345 - Delivery to: Local DC"

Scenario 2: Multi-format Document Set

Processing a batch containing 50 emails and 20 PDFs about various shipments

Open Questions

  • Neo/Recall Integration: API vs direct DB access? REST, GraphQL, or message queue?
  • Database Schema: Final structure and required fields TBD
  • Processing Volume: Expected throughput? Real-time vs batch processing?
  • Instruction Taxonomy: Complete list of action types to be extracted?
  • Authentication: M365 OAuth flow, PDF source location/permissions?
  • Error Handling: What happens with unparseable documents?

Project Structure

unified-document-parser/
├── src/
│   ├── parsers/           # Email and PDF parsers
│   ├── correlators/       # Deduplication & correlation logic
│   ├── formatters/        # DB schema transformation
│   └── integrations/      # Neo/Recall API client
├── tests/                 # Unit and integration tests
└── docs/                  # Documentation and design docs

Next Steps

  1. Define detailed DB schema for target system
  2. Establish Neo/Recall integration method
  3. Build unified parser interface and base classes
  4. Implement M365 email parser with OAuth authentication
  5. Implement PDF parser with text/table extraction
  6. Develop deduplication engine with configurable strategies
  7. Create correlation engine with instruction extraction
  8. Build schema formatter with validation
  9. Integrate with Neo/Recall module
  10. Add comprehensive testing and error handling