Initial reactions to Mahou Shoujo Madoka Magica have mostly labeled Homura as Fate to Madoka's Nanoha, partly based on her confrontations with Madoka. However, Steven Den Beste speculates that Homura's remarks to Madoka are not a threat, but a sincere and well-meant warning. I agree with him; this is exactly how I read the first episode.

Unlike SDB I don't think we need to posit any peculiar unreality in the pleasant world that Madoka inhabits to explain this. One of the background themes of Nanoha was the costs that Nanoha paid for becoming a magical girl, both in social isolation from her friends and (to some degree) in how what she witnessed and did wound up affecting her. My belief is that Shinbo is returning to this theme in Magica.

Thus, I think that Homura is simply trying to tell Madoka that she should stay out of the magical girl stuff and stay herself for her own sake. If she takes Kyubei's offer, if she believes him, it will destroy her current life just as much as if she became a soldier in a combat zone (because if you look at it right, that's what magical girls are). I further theorize that Homura dislikes Kyubei because she feels he is preying on the innocent (and lying to them) as he does his recruiting.

You could criticize Homura for not coming out and saying this directly, but I expect that she has reasons. Besides, no young fool ever believes the veteran who says that war is hell; all they can see is the romance.

(Anime has an attraction to 'war is hell' shows in various genres; Gundam has done several, for example. Sometimes it's a side theme, sometimes it's front and center.)

PS: I'm assuming here that Homura also experienced the opening 'dream' sequence somehow and recognizes Madoka from it. It may even be why Homura has transferred into Madoka's school at this point.

Sidebar: on the unpleasantness of being a magical girl

If you look at it from a realist angle, being a magical girl is not a really pleasant thing. You're risking your life (and health) in secret combat, combat that may strike at any time and any place with little or no warning; you take on great stresses (from the combat) and vast social isolation (from the secrets that you can't tell anyone). You become a flake to your friends and a liar to your family; you smile to everyone in a false front no matter what you actually feel. And you know that your friends and family are probably in terrible danger that you can't tell them about.

This is not my original idea; I owe it to exposure to, among other things, fanfics like Sailor Nothing (I am linking as I do so that you can see the concise description).

Since it wants to be cheerful, almost all magical girl anime ignores this. Nanoha was unusual in that the issue appeared as a background issue and a side theme.