Position a rack in the top third of the oven and heat the oven to 425°F.
Heat the oil and butter in a 10-inch skillet over medium-high heat. When the butter has melted, add the mushrooms, sprinkle with salt, and cook, stirring occasionally, until softened and browned slightly, about 8 minutes. If the mushrooms release a lot of water, increase the heat to evaporate it. Toss in the ham and cook, stirring frequently, until it has browned a bit, 2 to 3 minutes. Remove from the heat, season with a little salt and pepper, and set aside to cool.
Put the cornstarch in a medium bowl. Whisking steadily, slowly pour in 1/2 cup of the milk, mixing until quite smooth. Whisk in the whole eggs and egg yolks, mixing again until smooth, and then gradually whisk in the rest of the milk, the cream, salt, and cayenne.
Butter or oil a 2-quart, 7x11-inch Pyrex dish or an 8x11-inch ceramic gratin dish. Spread the ham and mushroom mixture over the bottom of the pan. Scatter on the Gruyère and chives.
Whisk the batter in case it has settled and then pour it into the baking dish. Bake until the top is deep golden and the custard is set (insert a pick in the center to check; it should be softly set, neither liquid nor firm), 25 to 30 minutes. Let stand for 30 minutes before serving.
The batter can be made up to one day ahead and refrigerated, covered.