Fort key not opening door

Is there a time limit in which to do the fort? I completed one but the key didn't open the door. Maybe I took too long. Oh well. Had to leave it.