Failing to pick up all the blossom from fruit trees will do no harm during the short season they are on the ground but don't offer as extra.
The stones from the plum fruit contain cyanide and should be avoided.
The leaves, as they get older and wilt, produce a cyanic compound, so although your tortoise will come to no harm if he accidentally nibbles a few fallen leaves, it is wise to prevent him from doing so if possible.
Damson (Prunus domestica subsp. insititia). See Plum fruit.