While progress in data-efficient AI offers a route to developing AI tools without relying on access to large volumes of training data, an increasing amount of data continues to be generated and captured from everyday activities. Removing names or obvious personal details is no longer an effective strategy for privacy preservation, in the context of AI algorithms that can draw connections between personal details available online or embedded in data. A variety of policy questions follow, from how to disrupt the market dominance of those companies with pre-existing access to data resources, to how to ensure the right to privacy of individuals amidst changing patterns of data use.