Oliver put Borough into the lead in the opening stages before hot-shot Iyseden Christie equalised shortly after interval with his eighth strike of the campaign.
Boyd, however, clinched a hard-earned victory for Borough in the 55th minute to end a promising sequence by Kidderminster, who had lost only once in their previous six league games.
Rob Davies, making his Harriers home debut following a loan move from West Bromwich Albion, was soon in the limelight when Mark Yates' men gained two corners inside the opening three minutes. The midfielder floated in both kicks, but they came to nothing.
Hard-working Stevenage, however, snatched the lead in the eighth minute when they capitalised on their first corner. It was taken by Steve Guppy and headed past goalkeeper Scott Bevan from close range by unchallenged defender Oliver.
Kidderminster wasted little time in trying to get back on level terms with a couple of determined efforts. Firstly, striker James Constable, making his debut after moving to Aggborough on loan from Walsall, went close with a volley and Gavin Hurren headed wide after connecting to a menacing cross from Jeff Kenna.
Borough full-back Barry Fuller needed treatment when he was injured in a fierce challenge by Russ Penn who gained midfield possession before whipping in a strong shot which went wide.
The unlucky Fuller, however, lasted only a few more minutes before he had to be replaced by Hasim Deen.
Stevenage's rearguard had to be increasingly on their guard as the first half progressed with Harriers battling hard to try to make a breakthrough.
Kidderminster's leading marksman Christie went close with a shot before Constable hooked wide after latching onto a header into the danger zone by Hurren.
Borough, however, were always alert up front and a promising breakaway in the last minute of the opening period resulted in under-pressure striker Steve Morison prodding a shot just off target.
Harriers took just two minutes of the second half to equalise with a splendidly taken goal by on-loan Christie. He collected a cross from Michael Blackwood before turning his marker Ronnie Henry and cracking in a 15-yard angled shot into the roof of the net.
Borough, however, were back in the lead when an attack involving Adam Miller and Guppy led to Boyd being given enough time to knock the ball past Bevan.
Shortly afterwards Stevenage had midfielder Bradley Johnson booked for a foul on Hurren which resulted in a free-kick by Davies flying high over the bar from in front of goal.
Boyd then tried his luck with a powerful drive which fizzed just wide before Christie bundled the ball into the net, but the referee's whistle had already gone for an infringement.
Bevan then denied Boyd before Stevenage were heavily indebted to goalkeeper Alan Julian who pulled off a terrific one-handed diving save to thwart Constable who was then prevented from scoring by a last-ditch block by Deen.
Yates drafted on front-runner Andy White and midfielder Brian Smikle in the closing stages in an unsuccessful bid to crack the solid Stevenage defence.
In the five tense minutes of stoppage time Julian was called on to pull off a fine save to stop a header from Mark Creighton following a cross by Simon Russell.