Versatile-OCR-Program: An OCR Tool That Extracts Structured Data From Complex Materials Like Exam Papers 58

What is Versatile-OCR-Program?

Versatile-OCR-Program is an OCR system specifically optimized for machine learning training. It can extract structured data from complex educational materials (such as exam papers) and output them in JSON or Markdown format. It supports Japanese, Korean, and English, and can be extended to other languages, mathematical formulas, tables, charts, and diagrams. It can also perform semantic annotation, generating natural language descriptions.

Main Features of Versatile-OCR-Program

• Optimized for Machine Learning Training: Extracted elements (such as charts, tables, and figures) are semantically annotated, including the automatic generation of natural language descriptions of visual content, to enhance downstream model training.

• Multilingual Support: Supports Japanese, Korean, and English, and can be easily customized for other languages.

• Structured Output: Generates ai-ready output in JSON or Markdown format, including human-readable descriptions of mathematical expressions, table summaries, and figure captions.

• High Accuracy: Achieves an accuracy of 90% - 95% on real-world academic datasets such as EJU Biology and Tokyo University Mathematics.

• Complex Layout Support: Capable of accurately handling exam-style PDFs containing dense scientific content, formula-heavy paragraphs, and rich visual elements.

• Technical Implementation: Uses technologies such as DocLayout-YOLO, Google Vision API, Gemini Pro Vision, MathPix OCR, OpenAI API, and OpenCV.

Sample Output

• Provides actual sample outputs, including input images and converted output images of math and biology questions, along with semantic context and extracted data in English translation.

• Sample Math Question: Demonstrates how to extract information from an image containing geometric figures and angles, and generate related descriptions and questions.

• Sample Biology Question: Demonstrates how to extract information from an image showing the process of mitosis in onion root tip cells, and generate related descriptions and questions.

Workflow of Versatile-OCR-Program

1. Step 1: Initial OCR Extraction: Run the ocr_stage1.py script to extract raw elements (text, tables, figures, etc.) from the input PDF, perform layout detection, and store intermediate results.

2. Step 2: Semantic Interpretation and Final Output: Run the ocr_stage2.py script to process the intermediate data and convert it into structured, human-readable output, including generating natural language explanations and summaries, and organizing the content into AI-ready formats (JSON/Markdown).

Technical Implementation

• Optimized Table Processing: Uses DocLayout-YOLO to detect table regions, uses Google Vision OCR for table processing (which works better for Japanese text), and preserves table structure in a structured JSON format.

• Image and Special Region Processing: Uses the image analysis feature of Google Vision API to process image regions, uses Google Cloud Vision API to generate image descriptions, uses the document analysis feature of Google Cloud Vision API to process charts/diagrams, and stores the processing results in a structured JSON format.

Input Effect

Output Effect

Applications of Versatile-OCR-Program

• Education Sector: Used to extract and organize data from educational materials such as exam papers and academic papers, providing support for educational research and learning.

• Machine Learning Training: Generates high-quality training datasets for training machine learning models, particularly for applications in the education sector.

• Multilingual Document Processing: Suitable for scenarios that require processing documents in multiple languages, such as international academic exchanges and digitization of multilingual educational materials.

github project:https://github.com/ses4255/Versatile-OCR-Program