Here's a design using a MOSFET to disconnect the two devices when Arduino power is absent:

^{simulate this circuit – Schematic created using CircuitLab}

When V2 (Arduino power) is low, Q1 is switched off. In that state, the I/O pin of "other" can have zero or greater potential, and no current (except negligible leakage) can pass via Q1's body diode.

When the Arduino has power, Q1's gate is held at +5V, and this design behaves like the popular level translator designs. Both devices have their inputs held high by default, due to R1 and R2. If either device's I/O pin becomes a low impedance output, and forces that pin low (0V), the MOSFET switches on, and the low state is duplicated on the other side.

Duplicate this R1, R2 and Q1 arrangement for the D3 and D4 channels.

Here's a design using a MOSFET to disconnect the two devices when Arduino power is absent:

^{simulate this circuit – Schematic created using CircuitLab}

When V2 (Arduino power) is low, Q1 is switched off. In that state, the I/O pin of "other" can have zero or greater potential, and no current (except negligible leakage) can pass via Q1's body diode.

When the Arduino has power, Q1's gate is held at +5V, and this design behaves like the popular bidirectional level translator. Both devices have their inputs held high by default, due to R1 and R2. If either device's I/O pin becomes a low impedance output, and forces that pin low (0V), the MOSFET switches on, and the other side adopts that same low potential.

Duplicate this arrangement of R1, R2 and Q1 for each of the other channels, D3 and D4.