Social Contract Theory by ChaosMotor (source)
Common misconception, the most common, really.
There is no "the" social contract, there is only your social contract and my social contract, each of which, in its own distinct but similar way, defines how each will interact with other members of society.
You're not "born into" a social contract, you create the social contract through your actions. It is your assurance to others how you will behave as proven by past behavior.
Most of us share largely similar to almost identical self-developed and self-expressed personal social contracts. Some people think they can take the most commonly agreed upon aspects of these myriad individual interpretations, in order to define when it is acceptable to use force against those who don't agree. That is how many justify government. "I will only use force on you that you would find justifiable to use on me," also known as "he deserved it".
So, you consent to the social contract to the extent that you are creating the social contract by your actions, and you must have consented to your own actions (else you would not have performed these acts, or you would have performed them under duress and thus they are not subject to the content of your existing social contract as they were not your actions, but the actions of another party, thrust upon you by force).
Thus we see there is no implied consent nor any dictates of society, it is simply a tool for informing society how you will interact with it.
Further, we see there is not necessarily a consent to the existence or actions of a state, because the state is built from our myriad social contracts, and our social contracts are not built from the state.
Statists would invert the pyramid and have you believe that the state is fundamental to the social contract, it is not; the social contract is the foundation upon which the state is built, and the state has no right to breach its boundaries and reach past the mutual consent of the governed, by want of their established boundaries for the state, based upon the mass of individuals' own interplay of social contracts.
Anti-statists would then refute the existence of any social contract, even those derived directly from the actions of the person whose social contract it is, by refuting the statist's inversion of the social contract which places the state at the fundament.
Both of these viewpoints are in error, because they require that the social contract be an external force, instead of an external representation of an internal state of each individual within society. In short, the social contract is most easily understood as a person's reputation, and expectation of their future actions.
Does this help resolve your concerns?
I think this is exactly how you misunderstand me, and how others misunderstand social contract theory. You need to understand from where the social contract is derived, and it's not derived from the outside of a person, it is not impressed upon them, it is derived from the inside of a person, and is exposed and explicated by how they behave.
Interaction does not require you to follow a supplied set of externally derived rules.
Interaction merely explicates for those you are interacting with, how they should expect you to interact with them. The manner in which you behave is informative to the other parties and used to develop their understanding of how you will behave. That is your social contract. How they respond to you is their social contract.
As it is in our best interests to align complimentary and mutually beneficial aspects of our individual social contracts, some find it useful to enshrine these as written "laws" which become the articles of the state. I dispute this as a necessity and find no utility in binding others to aspects of my social contract, or in being bound by others to aspects of their social contracts. I have the same refutations and problems with this idea, that you do.