Position a rack in the center of the oven and heat the oven to 350°F. Line a rimmed baking sheet with foil. Grease a 9x13-inch baking dish with the butter.
Pour the milk and cream into a small saucepan. Add the garlic, rosemary, and nutmeg. Bring just to a simmer, cover, and remove from the heat. Set aside to infuse for at least 20 minutes. Combine the Swiss cheese and Parmigiano in a bowl.
Peel the potatoes and, using a mandoline or your sharpest knife, slice them into 1/8-inch-thick rounds. Arrange about one-third of the potatoes in a single overlapping layer in the baking dish, season with kosher salt and white pepper, and top with one-third of the Swiss-Parmigiano mix. Scatter over half of the goat cheese. Add a second overlapping layer of potatoes, more salt and white pepper, another third of the Swiss- Parmigiano mix, and the remaining goat cheese. Make a third layer with the remaining potatoes and season with salt and white pepper. Press down lightly to compact the layer
Remove the garlic and rosemary from the infused cream, and discard them. Set the cream over medium-high heat and watch carefully until it just begins to simmer; don’t let it boil. Pour the cream over the potatoes and sprinkle the remaining cheese on top.
Set the baking dish on the foillined baking sheet, and bake until the top is deeply brown and the potatoes are completely tender when poked with a skewer, 1-1/4 to 1-1/2 hours. Let sit for 10 to 15 minutes before serving.