An afternoon sitting indoors contains little visible movement. Your body is still breathing, circulating blood and maintaining its tissues. Those processes use energy.
Exercise is one use of energy, alongside the work that continues at rest. Food does not become unnecessary on a day without a workout.