README.md

---
library_name: transformers
language:
- fr
- de
license: gpl-3.0
tags:
- ocr
- bloomfilter
- unigram
- impresso
- quality-assessment
- v1.0.6
---

# Model Card for `impresso-project/ocr-quality-assessor-unigram-light`

## Overview

This model is a **lightweight OCR quality assessor** for historical French and German texts. It is a streamlined version of the original [`impresso-project/OCR-quality-assessment-unigram`](https://huggingface.co/impresso-project/OCR-quality-assessment-unigram), now accessible via a Hugging Face `pipeline` for convenient integration into downstream tasks.

It uses **Bloom filters** containing known word unigrams to evaluate text quality by measuring the proportion of known vs. unknown words in OCR outputs. It is part of the [Impresso Project](https://impresso-project.ch), which develops tools for media archive processing and exploration.

## Model Details
### Model Description

- **Developed by:** University of Zurich (UZH) from the [Impresso team](https://impresso-project.ch). The project is an interdisciplinary project focused on historical media analysis across languages, time, and modalities. Funded by the Swiss National Science Foundation ([CRSII5_173719](http://p3.snf.ch/project-173719), [CRSII5_213585](https://data.snf.ch/grants/grant/213585)) and the Luxembourg National Research Fund (grant No. 17498891).
- **Model type:** Bloom filter–based scoring via a Transformers-compatible pipeline
- **Languages:** French (fr), German (de)
- **License:** GPL-3.0
- **Base resource:** [`impresso-project/OCR-quality-assessment-unigram`](https://huggingface.co/impresso-project/OCR-quality-assessment-unigram)
- **Interface:** Hugging Face `transformers` pipeline
- **Input format:** Raw text string
- **Output format:** Float score representing OCR quality

## How to Use

```python
from transformers import pipeline

MODEL_NAME = "impresso-project/ocr-quality-assessor-unigram-light"

ocrqa_pipeline = pipeline("ocr-qa-assessment", model=MODEL_NAME, 
                          trust_remote_code=True, 
                          device='cpu')

sentence = """En l'an 1348, au plus fort des ravages de la peste noire à travers l'Europe,
          le Royaume de France se trouvait à la fois au bord du désespoir et face à une opportunité."""

score = ocrqa_pipeline(sentence)
print(score)
```

## Output Format

Returns a single float value indicating the proportion of known tokens:

```python
{'ocr_quality_score': 0.76}
```

## Use Cases

- OCR pipeline evaluation and quality diagnostics
- Automated scoring of OCR segments or lines
- Quick feedback in web-based transcription and correction tools

## Dataset and Preprocessing

The Bloom filters used internally are derived from:
- Wikipedia dumps (historical and modern)
- Impresso-specific lexical resources

Text normalization includes:
- Unicode NFKC normalization
- Digit masking (0)
- Punctuation and symbol removal
- Lowercasing

## Limitations

- Currently supports only **French** and **German**
- Does not provide spell correction suggestions
- False positives are possible (due to the nature of Bloom filters)
- Quality score is approximate and works best at the **segment** or **line** level

## Environmental Impact

- **Hardware:** Standard laptop / CPU inference
- **Training:** Reuse of existing Bloom filters; minimal additional compute
- **Estimated Emissions:** < 0.01 kg CO₂eq

## Contact

- Website: [https://impresso-project.ch](https://impresso-project.ch)

<p align="center">
  <img src="https://github.com/impresso/impresso.github.io/blob/master/assets/images/3x1--Yellow-Impresso-Black-on-White--transparent.png?raw=true" width="300" alt="Impresso Logo"/>
</p>