When Dorian confronts Sibyl what effect does it have on his painting?

His painting had changed to display dispair and age. Specifically, there was a change around the mouth. “He had uttered a mad wish that he himself might remain young, and the portrait grow old; that his own beauty might be untarnished, and the face on the canvas bear the burden of his passions and his sins; that the painted image might be seared with the lines of suffering and thought, and that he might keep all the delicate bloom and loveliness of his then just conscious boyhood. Surely his wish had not been fulfilled? Such things were impossible. It seemed monstrous even to think of them. And, yet, there was the picture before him, with the touch of cruelty in the mouth.”

What has truly caused Sibyl to quit or act badly?

She has revealed who she really is. When Sibyl was acting, it was the only thing that she knew. Now that she has really found love, she has also found herself, which happens not to be an actor.