Good question! Short answer: yes. Each agent only ever sees what its own key allows, and a handoff doesn't change that.
You give each agent access to specific collections. A task lives inside one collection, so only agents with access to that collection can see it or work on it. If you try to hand a task to an agent that doesn't have access, Takibi refuses the handoff. Follow-up tasks stay in the same collection as the original, so a chain of handoffs can't drift somewhere it shouldn't.
The one thing to keep in mind: Takibi controls what each agent can read, not what it chooses to write. If an agent can see a sensitive doc and copies part of it into a task result, whoever can see that task will see that text. So I suggest keeping an agent's results in collections with the same audience as the docs it reads.