Actually, these type of dialog boxes shouldn't appear with .NET applications. It should be a dialog box with a detailed error description and a white cross on red circle. I guess something special must have happended here, so that this dialog appeared. As you say that the dialog appeared after you already could choose the resolution, I agree that it should be somehow related to the graphics. As BVE initializes DirectX probably after this dialog, I would give (re)installing DirectX a shot. BVE seems to use DirectX 7.
And if that doesn't help, click CANCEL in the dialog and post the debug information given there (if any).