"Agent after agent is leaving. No one feels the place has serious longevity," an insider told Page Six.