Swann and Hilton didn’t go because of fans being critical, harsh or negative. Swann had no money to spend in the end, after gambling it on the club and with his own personal problems, while Hilton never really had the money to begin with and used the “fans are driving me away” thing as a guilt trip to keep his supporters on side and pass off the blame.
