Position a rack in the center of the oven and heat the oven to 425°F. Sprinkle the steaks with the chopped rosemary, 1-1/2 tsp. salt, and 3/4 tsp. pepper. In a large (12-inch), oven-proof, heavy-duty skillet (like a cast-iron pan) over medium-high heat, combine 1 Tbs. each of the butter and oil and heat until the butter melts and its foam subsides. Add the steaks and cook without touching until they brown around the edges and easily release from the pan, about 3 minutes. Flip and cook the other sides until they brown, about 2 minutes. Transfer to a large plate.
Add the remaining 1 Tbs. oil to the skillet. Arrange the potatoes, cut side down, in the skillet, scatter the rosemary leaves around the potatoes, sprinkle with 1/2 tsp. salt, and transfer to the oven. Roast until the potatoes are tender when pierced, about 15 minutes. Set the steaks atop the potatoes and cook until the steaks are medium-rare—an instant-read thermometer inserted into the thickest part should register 135°F, about 10 minutes.
Transfer the steaks to a cutting board and let rest for 5 minutes. Toss the potatoes with the remaining 2 Tbs. butter and the balsamic vinegar, and keep warm in the oven (turn the oven off and leave the door slightly open). Serve the steaks thinly sliced, with the potatoes on the side, drizzled with any remaining balsamic and butter mixture and sprinkled lightly with salt and pepper.