Yes, Olympic left New York. for the last time on 5 April 1935, but she wasn't passengerless. Among those on board were Alfred Byrne, Lord Mayor of Dublin, and noted chess player Samuel Reshevsky.
At that point it was expected that she would make a number of cruises from New York over the summer, and some Atlantic crossings in the fall, but these plans were later scrapped, as was Olympic herself. She had not been withdrawn from service as of her 5 April New York departure.
Were you thinking of Berengaria, maybe? She sailed east without passengers in March 1938 after a fire at New York led U.S. authorities to revoke her certificate and she never sailed again.