The visitors won the shootout after an enthralling match in Sarajevo finished 1-1 after extra time.
Liam Boyce scored the winning penalty for Ian Baraclough's men after Bailey Peacock-Farrell saved from Haris Hajradinovic and Edin Visca missed.
Niall McGinn equalised for NI after Rade Krunic had opened the scoring.
Northern Ireland will now play Slovakia in the play-off final in November after they defeated the Republic of Ireland on penalties in Bratislava.