Yes, Olympic left New York. for the last time on 5 April 1935, but she wasn't passengerless. Among those on board were Alfred Byrne, Lord Mayor of Dublin, and noted chess player Samuel Reshevsky.
At that point it was expected that she would make a number of cruises from New York over the summer, and some Atlantic crossings in the fall, but these plans were later scrapped, as was Olympic herself. She had not been withdrawn from service as of her 5 April New York departure.