Despite a mercurial performance from Toni Kroos and an early headed goal from Shkodran Mustafi, they were troubled at the back at times.
Jerome Boateng cleared off the line superbly while keeper Manuel Neuer made several good saves before a late scare.
Yevhen Konoplyanka threatened for Ukraine but they tired, with Bastian Schweinsteiger adding a late second.
The Manchester United midfielder, who has not played since 20 March because of a knee injury, came off the bench in stoppage time and two minutes later scored a sweet half-volley from a swift counter attack.
And there were times when they looked vulnerable against Ukraine, who have never previously beaten them in five attempts.
Germany could have added to their early lead, which came from a debatable free-kick after Yaroslav Rakitskiy appeared to get the ball when tackling Thomas Muller.
Yet Neuer was kept busy in the German goal and Boateng made a brilliant goal-line clearance after he diverted Konoplyanka’s effort towards his own net.
Up front, Mario Gotze was isolated, and had the fewest touches of any player on the pitch. Fellow midfielders Muller and Mesut Ozil were not at their best, although the Arsenal midfielder crossed for Schweinsteiger’s well-taken goal.
Joachim Low’s side became more resolute at the back as the game wore on, and there is plenty of time for the team to find top form as the tournament progresses, which is a worrying thought for other teams.
Like fellow Real Madrid midfielder Luka Modric, who was influential in Croatia’s victory over Turkey earlier on Sunday, Kroos played like a man full of confidence after winning the Champions League last month.
Not only did he boss the game from the start in open play, his set-pieces were also deadly as he curled the ball in superbly for Mustafi to head in after 19 minutes.
Another brilliant pass from the former Bayern Munich player should have led to Germany doubling their lead but Sami Khedira’s shot was straight at Ukraine goalkeeper Andriy Pyatov.
The 26-year-old hit the corner of the post and crossbar early in the second half and showed plenty of evidence that his performances will be integral to the success of Germany in France.
They had the better opportunities to score in the first half and must have wished they could still call on the services of Andriy Shevchenko, who scored their only European Championship goals four years ago and is now part of Mykhailo Fomenko’s coaching staff.
Once Rakitskiy tested Neuer with a free-kick after the break, Ukraine’s inroads towards the German goal dried up. They almost equalised near the end when Mustafi headed over the advancing sweeper-keeper Neuer but Boateng got back quickly to thwart the danger.