This is the relevant page on French Telecom (Orange) site for mobile internet: Orange
It's quite a bit more expensive than in UK, where you can get 1GB download for £10 or so. Instead you buy a top-up card from 1 hour at 8 euro to 12 hours for 35 euro.
To get cheaper you have to take out at least a year's contract. 44.40 euro/month for up to 1GB (reduced speed beyond that) or 84.40 euro for up to 4GB. About 10% cheaper for a 2-year contract. There are also contracts that give free calls, text messages as wel as internet.
Actually, just about all the mobile companies seem to be offering those dongles (called "clé" here). You can check the websites for SFR and Bouygues in addition to Orange, but the pricing tends to be pretty uniform on these things.