Reseach Article

IntelliDocFormer: An Extended LayoutLMv3 Architecture with Graph-based Relational Reasoning and Document-Type-Aware Multimodal Fusion

by Gabriel Ihuoma Lilian, Laud Charles Ochei, Uche Oghenekaro
International Journal of Applied Information Systems
Foundation of Computer Science (FCS), NY, USA
Volume 13 - Number 5
Year of Publication: 2026
Authors: Gabriel Ihuoma Lilian, Laud Charles Ochei, Uche Oghenekaro
10.5120/ijaisdc016f874c3d

Gabriel Ihuoma Lilian, Laud Charles Ochei, Uche Oghenekaro . IntelliDocFormer: An Extended LayoutLMv3 Architecture with Graph-based Relational Reasoning and Document-Type-Aware Multimodal Fusion. International Journal of Applied Information Systems. 13, 5 ( Nov 2026), 68-77. DOI=10.5120/ijaisdc016f874c3d

@article{ 10.5120/ijaisdc016f874c3d,
author = { Gabriel Ihuoma Lilian, Laud Charles Ochei, Uche Oghenekaro },
title = { IntelliDocFormer: An Extended LayoutLMv3 Architecture with Graph-based Relational Reasoning and Document-Type-Aware Multimodal Fusion },
journal = { International Journal of Applied Information Systems },
issue_date = { Nov 2026 },
volume = { 13 },
number = { 5 },
month = { Nov },
year = { 2026 },
issn = { 2249-0868 },
pages = { 68-77 },
numpages = {9},
url = { https://www.ijais.org/archives/volume13/number5/intellidocformer-an-extended-layoutlmv3-architecture-with-graph-based-relational-reasoning-and-document-type-aware-multimodal-fusion/ },
doi = { 10.5120/ijaisdc016f874c3d },
publisher = {Foundation of Computer Science (FCS), NY, USA},
address = {New York, USA}
}
%0 Journal Article
%1 2026-11-16T21:32:16.026432+05:30
%A Gabriel Ihuoma Lilian
%A Laud Charles Ochei
%A Uche Oghenekaro
%T IntelliDocFormer: An Extended LayoutLMv3 Architecture with Graph-based Relational Reasoning and Document-Type-Aware Multimodal Fusion
%J International Journal of Applied Information Systems
%@ 2249-0868
%V 13
%N 5
%P 68-77
%D 2026
%I Foundation of Computer Science (FCS), NY, USA
Abstract

The digitisation of finance, healthcare, legal services and academia has produced large volumes of unstructured and semi-structured documents, from invoices and forms to scientific papers and contracts. Extracting information from them requires systems that understand both textual content and visual layout. Transformer-based models such as LayoutLM, DocFormer and Donut process documents multimodally, but they reason weakly about relations between spatially separated yet semantically related elements, treat all document types uniformly, have used visual encoders designed for natural scenes, and are computationally demanding. This paper presents IntelliDocFormer, an extended LayoutLMv3-based architecture that adds a Document-Type Classification (DTC) module, graph-based relational reasoning with a Graph Attention Network over a sparse layout graph, a Vision Transformer with Feature Pyramid Network (ViT-FPN) visual encoder, and an adaptive multimodal fusion layer. The architecture is analysed against existing models and evaluated with the Architecture Trade-off Analysis Method (ATAM) using a résumé-screening scenario extended to a utility tree of seven quality-attribute scenarios covering accuracy, throughput, degraded input, long documents, modifiability, explainability and fairness. The analysis shows that the modular design addresses document heterogeneity and complex layouts, and it identifies the sensitivity points, trade-offs and risks of the design, including the retained quadratic cost of the backbone. The contribution is conceptual and architectural; an implementation and validation plan on DocBank, FUNSD, CORD and DocVQA is specified for future empirical work.

References
  1. Appalaraju, S., Jasani, B., Kota, B.U., Xie, Y. and Manmatha, R. 2021 DocFormer: End-to-end transformer for document understanding. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). 993-1003.
  2. Xu, Y., Li, M., Cui, L., Huang, S., Wei, F. and Zhou, M. 2020 LayoutLM: Pre-training of text and layout for document image understanding. Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 1192-1200.
  3. Kim, G., Hong, T., Yim, M., Nam, J., Park, J., Yim, J., Hwang, W., Yun, S., Han, D. and Park, S. 2022 OCR-free document understanding transformer. Proceedings of the European Conference on Computer Vision (ECCV). 498-517.
  4. Huang, Y., Lv, T., Cui, L., Lu, Y. and Wei, F. 2022 LayoutLMv3: Pre-training for document AI with unified text and image masking. Proceedings of the 30th ACM International Conference on Multimedia. 4083-4091.
  5. Lee, C.-Y., Li, C.-L., Dozat, T., Perot, V., Su, G., Hua, N., Ainslie, J., Wang, R., Fujii, Y. and Pfister, T. 2022 FormNet: Structural encoding beyond sequential modeling in form document information extraction. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (ACL). 3735-3754.
  6. Ren, S., He, K., Girshick, R. and Sun, J. 2015 Faster R-CNN: Towards real-time object detection with region proposal networks. Advances in Neural Information Processing Systems (NeurIPS). 28: 91-99.
  7. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł. and Polosukhin, I. 2017 Attention is all you need. Advances in Neural Information Processing Systems (NeurIPS). 30: 5998-6008.
  8. Beltagy, I., Peters, M.E. and Cohan, A. 2020 Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150.
  9. Smith, R. 2007 An overview of the Tesseract OCR engine. Proceedings of the Ninth International Conference on Document Analysis and Recognition (ICDAR). 629-633.
  10. Devlin, J., Chang, M.-W., Lee, K. and Toutanova, K. 2019 BERT: Pre-training of deep bidirectional transformers for language understanding. Proceedings of NAACL-HLT 2019. 4171-4186.
  11. Xu, Y., Xu, Y., Lv, T., Cui, L., Wei, F., Wang, G., Lu, Y., Florencio, D., Zhang, C., Che, W., Zhang, M. and Zhou, L. 2021 LayoutLMv2: Multi-modal pre-training for visually-rich document understanding. Proceedings of ACL-IJCNLP 2021. 2579-2591.
  12. Powalski, R., Borchmann, Ł., Jurkiewicz, D., Dwojak, T., Pietruszka, M. and Pałka, G. 2021 Going full-TILT boogie on document understanding with text-image-layout transformer. Proceedings of the International Conference on Document Analysis and Recognition (ICDAR). 732-747.
  13. Li, C., Bi, B., Yan, M., Wang, W., Huang, S., Huang, F. and Si, L. 2021 StructuralLM: Structural pre-training for form understanding. Proceedings of ACL-IJCNLP 2021. 6309-6318.
  14. Gemelli, A., Biswas, S., Civitelli, E., Lladós, J. and Marinai, S. 2022 Doc2Graph: A task agnostic document understanding framework based on graph neural networks. Computer Vision – ECCV 2022 Workshops. Springer. 329-344.
  15. Tang, Z., Yang, Z., Wang, G., Fang, Y., Liu, Y., Zhu, C., Zeng, M., Zhang, C. and Bansal, M. 2023 Unifying vision, text, and layout for universal document processing. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 19254-19264.
  16. Wang, D., Raman, N., Sibue, M., Ma, Z., Babkin, P., Kaur, S., Pei, Y., Nourbakhsh, A. and Liu, X. 2024 DocLLM: A layout-aware generative language model for multimodal document understanding. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL). 8529-8548.
  17. Lv, T., Huang, Y., Chen, J., Cui, L., Ma, S., Chang, Y., Huang, S., Wang, W., Dong, L., Luo, W. et al. 2023 Kosmos-2.5: A multimodal literate model. arXiv preprint arXiv:2309.11419.
  18. Ye, J., Hu, A., Xu, H., Ye, Q., Yan, M. et al. 2023 mPLUG-DocOwl: Modularized multimodal large language model for document understanding. arXiv preprint arXiv:2307.02499.
  19. Blecher, L., Cucurull, G., Scialom, T. and Stojnic, R. 2023 Nougat: Neural optical understanding for academic documents. arXiv preprint arXiv:2308.13418.
  20. Chalkidis, I., Fergadiotis, M., Malakasiotis, P., Aletras, N. and Androutsopoulos, I. 2020 LEGAL-BERT: The muppets straight out of law school. Findings of the Association for Computational Linguistics: EMNLP 2020. 2898-2904.
  21. Li, M., Xu, Y., Cui, L., Huang, S., Wei, F., Li, Z. and Zhou, M. 2020 DocBank: A benchmark dataset for document layout analysis. Proceedings of the 28th International Conference on Computational Linguistics (COLING). 949-960.
  22. Veličković, P., Cucurull, G., Casanova, A., Romero, A., Liò, P. and Bengio, Y. 2018 Graph attention networks. Proceedings of the 6th International Conference on Learning Representations (ICLR). Vancouver, Canada.
  23. Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J. and Houlsby, N. 2021 An image is worth 16x16 words: Transformers for image recognition at scale. Proceedings of the 9th International Conference on Learning Representations (ICLR).
  24. Lin, T.-Y., Dollár, P., Girshick, R., He, K., Hariharan, B. and Belongie, S. 2017 Feature pyramid networks for object detection. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2117-2125.
  25. Hendrycks, D. and Gimpel, K. 2016 Gaussian error linear units (GELUs). arXiv preprint arXiv:1606.08415.
  26. Smock, B., Pesala, R. and Abraham, R. 2022 PubTables-1M: Towards comprehensive table extraction from unstructured documents. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 4634-4642.
  27. Yu, W., Lu, N., Qi, X., Gong, P. and Xiao, R. 2020 PICK: Processing key information extraction from documents using improved graph learning-convolutional networks. Proceedings of the 25th International Conference on Pattern Recognition (ICPR). 4363-4370.
  28. Dobrica, L. and Niemelä, E. 2002 A survey on software architecture analysis methods. IEEE Transactions on Software Engineering. 28(7): 638-653.
  29. Kazman, R., Bass, L., Abowd, G. and Webb, M. 1994 SAAM: A method for analyzing the properties of software architectures. Proceedings of the 16th International Conference on Software Engineering (ICSE). 81-90.
  30. Kazman, R., Klein, M. and Clements, P. 2000 ATAM: Method for architecture evaluation. Technical Report CMU/SEI-2000-TR-004. Software Engineering Institute, Carnegie Mellon University, Pittsburgh.
  31. Bass, L., Clements, P. and Kazman, R. 2012 Software Architecture in Practice, 3rd ed. Addison-Wesley, Boston.
  32. European Union 2024 Regulation (EU) 2024/1689 laying down harmonised rules on artificial intelligence (Artificial Intelligence Act). Official Journal of the European Union. L series, 12 July 2024.
  33. European Union 2016 Regulation (EU) 2016/679 (General Data Protection Regulation). Official Journal of the European Union. L 119: 1-88.
  34. Zhong, X., Tang, J. and Jimeno Yepes, A. 2019 PubLayNet: Largest dataset ever for document layout analysis. Proceedings of the International Conference on Document Analysis and Recognition (ICDAR). 1015-1022.
  35. Harley, A.W., Ufkes, A. and Derpanis, K.G. 2015 Evaluation of deep convolutional nets for document image classification and retrieval. Proceedings of the 13th International Conference on Document Analysis and Recognition (ICDAR). 991-995.
  36. Jaume, G., Ekenel, H.K. and Thiran, J.-P. 2019 FUNSD: A dataset for form understanding in noisy scanned documents. Proceedings of the ICDAR Workshop on Open Services and Tools for Document Analysis (ICDAR-OST). 1-6.
  37. Park, S., Shin, S., Lee, B., Lee, J., Surh, J., Seo, M. and Lee, H. 2019 CORD: A consolidated receipt dataset for post-OCR parsing. Document Intelligence Workshop at NeurIPS 2019. Vancouver, Canada.
  38. Mathew, M., Karatzas, D. and Jawahar, C.V. 2021 DocVQA: A dataset for VQA on document images. Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV). 2200-2209.
  39. Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G. et al. 2019 PyTorch: An imperative style, high-performance deep learning library. Advances in Neural Information Processing Systems (NeurIPS). 32: 8024-8035.
  40. Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A. et al. 2020 Transformers: State-of-the-art natural language processing. Proceedings of EMNLP 2020: System Demonstrations. 38-45.
  41. Fey, M. and Lenssen, J.E. 2019 Fast graph representation learning with PyTorch Geometric. ICLR Workshop on Representation Learning on Graphs and Manifolds. New Orleans, USA.
Index Terms

Computer Science
Information Sciences

Keywords

Document parsing Data structuring Transformer architecture LayoutLMv3 Multimodal fusion Graph neural networks Document-type classification ATAM