Processing Pipeline

graph LR subgraph "Input Stage" URLs[Article URLs] Config[Processing Config] OutputDir[Output Directory] end subgraph "Fetch Stage" HttpRequest[HTTP Request] RateLimit[Rate Limiting] SessionPool[Session Pooling] RetryLogic[Retry Logic] ResponseValidation[Response Validation] end subgraph "Parse Stage" HTMLParser[HTML Parser] ContentExtraction[Content Extraction] MetadataExtraction[Metadata Extraction] LinkProcessing[Link Processing] CleanupHTML[HTML Cleanup] end subgraph "Media Stage" MediaDetection[Media Detection] URLExtraction[URL Extraction] TypeClassification[Type Classification] MediaDownload[Media Download] ImageProcessing[Image Processing] FileOrganization[File Organization] end subgraph "Conversion Stage" MarkdownConversion[Markdown Conversion] LinkUpdating[Link Updating] ImageEmbedding[Image Embedding] ContentStructuring[Content Structuring] MetadataInsertion[Metadata Insertion] end subgraph "HTML Generation Stage" TemplateLoading[Template Loading] ContentRendering[Content Rendering] StyleApplication[Style Application] NavigationGeneration[Navigation Generation] AssetLinking[Asset Linking] end subgraph "Output Stage" DirectoryCreation[Directory Creation] FileWriting[File Writing] PermissionSetting[Permission Setting] ProgressTracking[Progress Tracking] ErrorLogging[Error Logging] end %% Flow connections URLs --> HttpRequest Config --> RateLimit OutputDir --> DirectoryCreation HttpRequest --> RateLimit RateLimit --> SessionPool SessionPool --> RetryLogic RetryLogic --> ResponseValidation ResponseValidation --> HTMLParser HTMLParser --> ContentExtraction ContentExtraction --> MetadataExtraction MetadataExtraction --> LinkProcessing LinkProcessing --> CleanupHTML CleanupHTML --> MediaDetection MediaDetection --> URLExtraction URLExtraction --> TypeClassification TypeClassification --> MediaDownload MediaDownload --> ImageProcessing ImageProcessing --> FileOrganization CleanupHTML --> MarkdownConversion FileOrganization --> LinkUpdating MarkdownConversion --> LinkUpdating LinkUpdating --> ImageEmbedding ImageEmbedding --> ContentStructuring ContentStructuring --> MetadataInsertion MetadataInsertion --> TemplateLoading TemplateLoading --> ContentRendering ContentRendering --> StyleApplication StyleApplication --> NavigationGeneration NavigationGeneration --> AssetLinking MetadataInsertion --> DirectoryCreation AssetLinking --> FileWriting DirectoryCreation --> FileWriting FileWriting --> PermissionSetting PermissionSetting --> ProgressTracking %% Error handling RetryLogic -.-> ErrorLogging MediaDownload -.-> ErrorLogging FileWriting -.-> ErrorLogging %% Styling classDef input fill:#e3f2fd classDef fetch fill:#e8f5e8 classDef parse fill:#fff3e0 classDef media fill:#fce4ec classDef convert fill:#f3e5f5 classDef html fill:#f1f8e9 classDef output fill:#ffe0b2 class URLs,Config,OutputDir input class HttpRequest,RateLimit,SessionPool,RetryLogic,ResponseValidation fetch class HTMLParser,ContentExtraction,MetadataExtraction,LinkProcessing,CleanupHTML parse class MediaDetection,URLExtraction,TypeClassification,MediaDownload,ImageProcessing,FileOrganization media class MarkdownConversion,LinkUpdating,ImageEmbedding,ContentStructuring,MetadataInsertion convert class TemplateLoading,ContentRendering,StyleApplication,NavigationGeneration,AssetLinking html class DirectoryCreation,FileWriting,PermissionSetting,ProgressTracking,ErrorLogging output