Heat the oven to 425°F. In a medium bowl, toss the shrimp with 1/2 Tbs. of the oil, and season with salt and pepper. Pour onto one side of a large rimmed baking sheet and arrange in a single layer. In the same bowl, toss the potatoes with another 1/2 Tbs. olive oil, 1/4 tsp. of the minced rosemary, and more salt and pepper. Pour onto the other side of the baking sheet and arrange in a single layer. Roast until the shrimp are cooked through and opaque, 7 to 8 minutes. Remove the shrimp from the oven and set aside to cool. Return the potatoes to the oven and continue to roast until tender and slightly crisp around the edges, about 15 minutes longer. When the shrimp are cool enough to handle, peel them and set aside.
Meanwhile, chop the garlic, add two good pinches salt, and mince to a coarse paste. In a small bowl, combine the minced garlic, lemon zest and juice, the remaining 1/4 tsp. rosemary, a pinch more salt, and a few grinds of pepper. Whisk in the remaining 3 Tbs. olive oil. Taste and add more lemon juice if needed.
When the potatoes are done, put them in a medium bowl, along with the shrimp, parsley, and olives. Add half the dressing; toss to coat. In another bowl, toss the arugula with the remaining dressing. Arrange the dressed arugula on dinner plates, top with the shrimp and potatoes, and serve.