This idea was given to me by a guest reviewer in regards to potential ideas, and I've decided to use the first idea. Someone else has also done something similar and I don't if others have as well, but I've decided to have my own crack at it.
Correction 9: Season 5 Episode 23 Entitled "Lian Yu"
As Oliver stood in the boat and clutched his son close to him after having rescued him from Chase, the latter sat silently across from them and observed their reunion. After a moment, he spoke.
"You won. Your son has his father back. And he learned exactly who his father was", he acknowledged while feeling a small bit of glee at that last part, and then added, "Just like you learned, who your father was right here, on these very same waters."
Oliver looked at him incredulously, mouth slightly open, as he listened to his words.
"What?", he asked the madman, wondering where he was going with this. Chase then continued.
"William's younger than you were, so he's going to be fine. And you have each other…"
"What are you saying?", Oliver asked him, feeling a pit of dread beginning to form in his stomach.
"Which is good", Chase continued. "Oliver that's good. Because it's gonna be lonely. Without mom, and Felicity", and then produced a gun and began lifting it to his forehead.
Eyes widening in horror, and realizing want his adversary was about to do, Oliver immediately leapt forward, grabbed the villain's arm and wrenched the gun from his hand and, without preamble, knocked his butt out.
He then quickly returned to his son's side and pulled him into his arms once more, relieved that a potential tragedy had been avoided. And a potential tragedy had been avoided since Samantha, William's mother, did not die, and after everyone was safe and sound, she moved to Star City and allowed Oliver to have a greater role in his son's life.
A/N And that is that. And the guest was right. Oliver did stand around like an idiot instead of stopping Adrian from killing himself.