Arabic text not rendered correctly in filled PDF forms after version 3.9.1

[almousa2000]

While testing PDF form filling with the latest version of pypdf, I noticed that Arabic text is not displayed correctly in the generated PDF.
However, when I downgrade to version 3.9.1, Arabic text renders properly.
The issue seems to have been introduced after that version.

Unfortunately, version 3.9.1 does not support filling dropdown fields, so I can’t use it as a workaround for all cases.

Environment

python 3.13.1

$ python -m platform
Windows-11-10.0.26100-SP0

$ python -c "import pypdf;print(pypdf._debug_versions)"
pypdf==6.1.3, crypt_provider=('cryptography', '46.0.3'), PIL=12.0.0

Code + PDF

This is a minimal, complete example that shows the issue:

from pypdf import PdfReader, PdfWriter

reader = PdfReader("input.pdf")
writer = PdfWriter()

writer.append(reader)

writer.update_page_form_field_values(
    writer.pages[0], {
        "Defendant full name":"احمد الموسى",
        "Defendant_email": "test@gmail.com",
        "Place": "Al Ain العين"
    }
)

with open("output.pdf", "wb") as output_stream:
    writer.write(output_stream)

input pdf:

input.pdf

output images:

Traceback

This is the complete traceback I see:

no error just not support

[stefan6419846]

Thanks for the report. You are of course invited to bisect this and propose a corresponding fix.

[PJBrs]

This is what Google Gemini says:

You've identified a common challenge when dealing with non-Latin scripts like Arabic in PDF text streams—the issue lies in the character encoding and the way the Arabic glyphs are shaped and ordered.

The part of the code that needs scrutiny for Arabic text is the encoding and display of the text string.

🛑 The Core Problem: Encoding for Arabic
The final part of the loop handles encoding and drawing:

Python

# 1. Map characters to bytes
encoded_line: list[bytes] = [
    font_glyph_byte_map.get(c, c.encode("utf-16-be")) for c in line
]
# 2. Draw the text
if any(len(c) >= 2 for c in encoded_line):
    ap_stream += b"<" + (b"".join(encoded_line)).hex().encode() + b"> Tj\n"
else:
    ap_stream += b"(" + b"".join(encoded_line) + b") Tj\n"

Issues with Standard Encoding for Arabic:
Bi-Directional Text (BiDi): Arabic is read right-to-left (RTL). The input text (and thus line) will be in logical order (the order you type or read it). PDF text operators (Tj) draw text in the order they are given, which must be the visual order (the order the characters appear on the page, which is usually right-to-left for Arabic). The input line must be reordered.

Contextual Shaping: Arabic characters change shape (glyph) based on their position in a word (initial, medial, final, isolated). The simple mapping font_glyph_byte_map.get(c, ...) assumes a single character c maps to a single byte sequence. For Arabic, c (a logical character) must first be shaped into its contextual form (glyph) before its byte-code equivalent is looked up.

UTF-16-BE Fallback: The fallback c.encode("utf-16-be") is correct for representing Unicode characters in a PDF text stream (often used with ToUnicode CMaps), but it doesn't handle the shaping or BiDi reordering.

🛠️ Proposed Solution to Fix Arabic Symbols
To correctly render Arabic, you need to introduce an explicit step for BiDi reordering and contextual shaping before the final byte-encoding. This often requires an external library (like python-bidi or relying on the operating system's text shaping engine), but assuming pypdf wants to keep core dependencies low, the requirement is to use correctly pre-shaped and reordered text.

1. Pre-Shaping Requirement
   For Arabic to display correctly, the text being passed into this method must be in visual order and use presentation forms (shaped glyphs).

If you control the calling code: The text argument needs to be processed before calling _generate_appearance_stream_data using a BiDi/Shaping library.

If you must fix it inside this method: You would need to import and use a library for shaping and reordering the text stream:

Python

# Assuming a necessary library is imported (e.g., Arabic-reshaper, python-bidi, etc.)
# This is a conceptual implementation of the required step:

from arabic_reshaper import reshape
from bidi.algorithm import get_display

# ... (code before the loop) ...

# Modify the initial text processing:
if not is_comb and not is_multiline and not font_glyph_byte_map:
    # If using a basic font and expected to display complex text, reorder and reshape
    # Note: This is an oversimplification; full shaping requires knowing font glyphs.
    # But for display streams, you often need the final visual string.
    
    # 1. Reshape characters (e.g., isol -> init -> medi -> final forms)
    reshaped_text = reshape(text)
    # 2. Reorder for visual (Right-to-Left) display
    visual_text = get_display(reshaped_text)
else:
    visual_text = text # Fallback for other cases
    
# ... (use visual_text in subsequent text processing logic) ...

# ... (inside the loop) ...

# The loop still processes the text based on line/char segments, but those segments
# must now be in the correct visual (RTL) order.

2. Simplifying the Character Encoding inside the Loop
   Assuming the input line is correctly ordered and uses presentation forms, the issue might also be the simple concatenation and encoding. Since the font is likely using a custom CMap (as discussed in previous turns), the logic should lean on the provided font_glyph_byte_map.

The existing loop encoding logic is mostly correct for combining bytes from the map:

Python

encoded_line: list[bytes] = [
    font_glyph_byte_map.get(c, c.encode("utf-16-be")) for c in line
]
if any(len(c) >= 2 for c in encoded_line):
    # Use hex encoding for multi-byte output (like UTF-16)
    ap_stream += b"<" + (b"".join(encoded_line)).hex().encode() + b"> Tj\n"
else:
    # Use parentheses for 1-byte output (like WinAnsi/Custom Encoding)
    ap_stream += b"(" + b"".join(encoded_line) + b") Tj\n"

If Arabic characters are still wrong, it confirms that the incoming line is not shaped or reordered appropriately for the final PDF rendering environment. The fix is external to this function's loop, lying in the text preparation or the font_glyph_byte_map creation.

For Arabic rendering, you must ensure that:

BiDi Reordering is applied to text (or lines).

Contextual Shaping is applied to convert logical characters to visual glyphs.

[PJBrs]

That garbled output you're seeing, ' - E / ' D E H 3, is a classic sign of incorrect Arabic text encoding and rendering within your PDF environment.

The string you provided, "احمد الموسى" (Ahmad Al-Mousa), is being processed character-by-character based on its visual representation in the PDF text stream, which is the sequence of the letters themselves, not the final rendered word.

Here's a breakdown of the problem and where the issue lies in the code's approach to the Arabic language:

💥 The Encoding Misinterpretation
Your output looks like it is displaying the individual bytes or character codes used for the text stream, but separated and possibly reversed, demonstrating the three major challenges with Arabic in PDFs:

1. Bi-Directional (BiDi) Reordering (The Main Issue)
   Arabic is written Right-to-Left (RTL). Your input string "احمد الموسى" is in logical order (the way you type it). PDF viewers expect text operators (Tj) to receive the characters in visual order.

Logical Order: ا (A) ح (H) م (M) د (D) ا (A) ل (L) م (M) و (W) س (S) ى (Y)

The PDF rendering engine is likely receiving the characters in an incorrect sequence, resulting in the jumbled appearance.

2. Contextual Shaping (Disconnected Characters)
   Arabic letters change shape based on their position in a word (e.g., isolated, initial, medial, final).

Correctly Shaped/Connected: احمد الموسى

Incorrectly Shaped/Isolated: ا ح م د ا ل م و س ى

The PDF requires the text to be passed using the byte-codes corresponding to the shaped glyphs, not the base logical characters. Your current text processing logic:

Python

font_glyph_byte_map.get(c, c.encode("utf-16-be")) for c in line
does not perform this essential shaping step. It looks up the byte for the base character c, which results in the glyphs being displayed in their isolated form (like your garbled output implies).

3. The Unwanted Characters (-, /, D, E, H, 3)
   These non-Arabic characters in your sample output (- E / ' D E H 3) are likely the result of the PDF reader attempting to use a default 8-bit encoding (like /WinAnsiEncoding) to interpret the high-byte-value data intended for the Arabic font, or a misinterpretation of multi-byte hex codes. It's a symptom of the encoding failure, not the root cause.

📝 Required Fix: External Text Processing
As identified previously, the fix requires an external step before the character encoding loop in _generate_appearance_stream_data.

To correctly render Arabic, the text or line variable must be transformed by a specialized library that performs:

Reshaping: Converts the logical characters (م) into their contextual forms (ـمـ, مـ, ـم).

BiDi Reordering: Reverses the text segments for correct RTL visual display.

You cannot fix this correctly with just the existing string and dictionary lookups; you must use a library capable of Arabic text processing (like python-bidi and arabic-reshaper) before passing the string to this encoding function.

Once the text is correctly shaped and reordered, the existing encoding logic will work as intended, mapping the presentation forms to their correct byte codes in the font's CMap.

[PJBrs]

The above is not the only problem. You also need to find the font resource that contains arabic script, which in the case of your pdf form is not straightforward. The current code is insufficient.

Really, this is not a regression, this was never supported, but it was invisible because, before, we didn't try to generate an appearance stream for text form fields.

[PJBrs]

Robust Matching Sequence
A comprehensive font matching strategy should always check the /FontDescriptor's /FontName:

Check /BaseFont: Get the name from the main Font Dictionary's /BaseFont entry (e.g., ABCDEF+ArialMT). Strip the 6-character tag prefix (ABCDEF+) for the first match attempt.

Check /FontDescriptor > /FontName: If the first match fails, navigate to the /FontDescriptor dictionary and check its /FontName entry. This is especially useful because the /BaseFont name can sometimes be omitted or simplified, while the /FontDescriptor name is mandatory if metrics are provided. Strip the tag prefix here as well for reliable matching against known font names.

Check Embedded File Data: For maximum robustness (as in iText), if both of the above fail, read the actual embedded font stream (/FontFile2 or /FontFile3) and parse the internal font table to find the true, un-mangled PostScript name.

[PJBrs]

... I'm trying to come up with a list of additions necessary to actually be able to produce an appearance stream from a text widget annotation with arabic characters:

• Extend the FontDescriptor so that it can parse embedded fonts
• Change the heuristic for identifying font resources for text annotations so that it can also match subsetted embedded fonts
• Make it possible to associate multiple font resources with one appearance stream, for instance when one font resource includes latin characters and another includes arabic characters (this is the case for the form in this example)
• For each word in the text in the form field, check which font resource to use, and add Tf operator accordingly
• For each word in the text in the form field, detect whether it is Right-To-Left and reorder characters accordingly
• For arabic characters, add a reshaping step for each word so that it is displayed correctly