I tested your code and it works fine with my test boards. So it must be a fault with the hardware. The fault you described sounds like a problem with the left shift register (U5) as this shift register deals with the LEDs and the Digits. It sounds like one mode is turning on all the outputs and the other mode turns off all the outputs (LEDs turn on when outputs are on and digits turn on when outputs are off). We should test the shift register for faults. Take the chip out, then bend the output pins up (pins 15 & 1-7). Now reseat the chip in the holder and upload the following code:

const int OE = 2;
const int SER = 11;
const int CLK = 12;
const int LAT = 13;

void setup() 
{
    pinMode(OE, OUTPUT); 
    pinMode(SER, OUTPUT); 
    pinMode(CLK, OUTPUT); 
    pinMode(LAT, OUTPUT); 

    analogWrite(OE, 0);
}

void loop() 
{

    digitalWrite(LAT,LOW);
    shiftOut(SER,CLK,LSBFIRST,B10000000);
    digitalWrite(LAT,HIGH);

}

Use your multimeter to check output 1 (pin 15) reads ~5V. Check the rest of the outputs read 0V. Now change the byte in the code to B0100000 and check the outputs. pin 15 should read 0V, pin 1 should read ~5V and the rest should read 0V. Keep changing the code and checking the readings. If anything looks wrong, try the other shift registers you have.

Other things to look at are the surface mount resistors and LEDs, which look a bit messy (well done for trying they are tricky). Also, try different ports for each connection, as all the connections to the Digitiser except the pot can be connected to any IO port (make sure you change the code to reflect changes).

Let me know what you find.