When the Air Conditioning system is running in your home, there are negative and positive pressure differentials put in the living area. In a perfect world the return air would seamlessly flow from any particular area through your system and back out through your ducts as conditioned air. This isn't a perfect world and many factors go into the actual path that air travels in your home.
Ideally each room would have a return sized for the flow of air in that area. This isn't normally practical and certain building standards have been developed over the years to maximize the efficiency of the system. The most typical home setup is that each room will have one or more ducts for the conditioned air, and then each floor will have at least one return duct, and then some larger rooms will have additional returns (family room, master bedroom, etc...)
Over the years the standards have changed as to what to do with the remaining rooms. In all but the newest homes HVAC designers simply relied on the gap under the door of each of the smaller rooms to transfer the air. Newer homes will often have vents above the door of each room that allow air to flow from one side of the wall to the other. Some homes have returns in every room (this is very rare, but not unheard of)
In houses that rely on the gap under the door, often carpet is upgraded or other new flooring installed and this gap is closed. It is often not a sufficient gap to begin with. If the room's doors are regularly closed this can lead to a relatively high pressure differential on the home.
What happens with the air?
Pressure will always attempt to equalize through the path of least resistance. Air is a fluid, and acts as such. The gaps in the flooring along the edges of the wall is a common place for this path to exist. As a result the air filters through the carpet in these areas, and all of the impurities in the air get left behind in the carpet.
Homes with indoor air quality issues (smoking, burning candles, incense etc...) will have more staining, whereas cleaner homes will have less. This doesn't necessarily mean that the air is dirty, it could also be that the carpet is just old. Changing the way your home's air flows can alleviate many of these problems. You could install the vents above the doors, seal flooring edges and baseboards, or just simply leave doors open.
Can filtration soiling be cleaned?
This can be very difficult to clean completely, and will often reappear. A commercial steam cleaning with aggressive pretreatment is your best option. Doing your own pre-treatment before a professional cleaning will likely yield the best results. Use a carpet cleaner/pre-treatment with a stiff bristled brush before the professional cleaners arrive. It is labor intensive. In the end, however, this may only be a temporary fix.
When replacing your carpet look at sealing the edges of the floor with caulk before installation. In particularly bad homes, the HVAC system may need additional intakes, or grates installed above the doors to allow proper air flow.
As always, I hope you have learned something here, and enjoy your home!