by Dr Nawab John Lone
AI photo apps may collect facial data beyond images, raising privacy concerns about biometric information, data retention, model training, and user consent, despite their entertainment appeal.
The latest wave of AI photo games and social media image tools sells itself as pure entertainment. Upload a picture, and the app hands back a cartoon avatar, a fantasy portrait, or a version of your face swapped into a movie scene.
Behind that instant transformation, though, sits a genuine data processing pipeline, one built to detect faces, map their geometry, and in some cases hold onto the results long after the joke has been shared and forgotten. The risk is not limited to selfies taken specifically for the app. A profile picture, an old family photo, or a casual holiday snapshot can carry enough facial information for a service to analyse identity related features, depending on how the app was built and what its privacy policy actually allows.
How these apps actually work
Most of these tools follow a similar pattern. The app uploads the photo to a server, runs a face detection model to find the face in the frame, identifies landmarks such as the eyes, nose, mouth and jawline, and then feeds that information into an AI model that generates the new image. In other words, the software is not simply painting over a photo. It is often converting the face into machine readable information first.
The Federal Trade Commission made this point explicit in a policy statement it adopted in May 2023. The agency defined biometric information broadly, covering data that depicts or describes a person’s physical, biological or behavioral traits, and it said the output of these systems, things like faceprints, embeddings and templates, can itself count as biometric information when it can reasonably be linked back to a person. That means a photo can be turned into a reusable numeric signature even when the only thing the user ever sees is a stylised cartoon.
What gets uploaded, and what can be inferred from it
These services generally need the original image file, and that file can carry more than the visible picture. Depending on how a photo was taken and shared, it may include metadata such as the time it was captured, the device model, and sometimes location. Once a photo reaches a server, it can also be analysed for facial geometry, estimated age, gaze direction, objects in the background, and other patterns that computer vision systems are trained to pick out.
A 2024 study from the University of Wisconsin-Madison illustrates how much can be pulled from a single image. A team led by Kassem Fawaz, an associate professor of electrical and computer engineering, examined the machine learning models built into the Instagram and TikTok apps and found that Instagram’s on-device model could infer several hundred different concepts from a photo, while TikTok’s model focused on estimating age and gender. The researchers also found that these models sometimes misclassified what they saw, raising concerns about bias if similar techniques were ever used for tasks like age verification.
Worth noting: in Instagram’s case, the researchers found that this particular analysis happened locally on the phone and was cached on the device itself, with no evidence in their testing that this specific cache was sent to Instagram’s servers. That is a useful reminder that “your photo was analysed” and “your photo was uploaded” are not always the same claim, though the swap and stylising apps this article is mainly concerned with typically do need to send the image to the cloud, since the finished result has to be generated somewhere.
Where the data goes after that
What happens once an image reaches a company’s server depends entirely on that company’s own policy. Some apps say they delete uploaded photos shortly after processing. Others keep them to operate the service, investigate abuse, run analytics, or improve their models. The exact wording matters a great deal here, because a promise to delete a photo can still leave room for backups, internal logs, or copies already used to train a separate system.
Two mainstream examples show how far this has already spread beyond novelty apps. Meta rolled out a batch of new AI features on Facebook in June 2026, including photo presets that let users change their clothing, hair or accessories with AI, a “Wear It” tool that lets people virtually wear a sports jersey, and camera roll sharing suggestions that scan a user’s device for photos worth turning into a collage or a video montage. Meta describes the camera roll suggestions as opt-in and says they can be turned off at any time.
Google Photos has gone through something similar with its Gemini-powered features, which the company says can draw on the photos and videos in a user’s library, on face group labels and names the user has assigned, and on account information, to power tools like Ask Photos and automatic title suggestions. Google says this data is not used for advertising and is not used to train generative AI models outside of Google Photos, though it can still be used to make inferences, such as estimating the age or general location associated with a face group.
What the policies usually say
Privacy policies typically grant a company broad permission to host, store, process, modify and create new works from whatever content a user uploads, all in the name of “providing the service.” Many also reserve the right to use that content for improving models, running analytics, protecting safety, personalising features, or supporting research. Users generally agree to this by accepting a terms of service page or tapping through an in-app prompt, and that consent can end up thin if the policy is written broadly or never clearly says whether an image will be kept, used to train a model, or tied to a permanent account record.
Why a face is not just a photograph?
A stylised, edited photo is just a picture with cosmetic changes applied to it. Facial recognition data is something else: it is information used to identify or verify a specific person, and once that information is captured, it typically falls into the legal category of biometric identifiers.
A facial embedding is different again. It is a string of numbers, generated by a machine learning model, that represents a face as a point in mathematical space. As the FTC’s 2023 policy statement makes clear, embeddings and faceprints can count as biometric information in their own right, which means a company can hold onto a reusable, searchable representation of someone’s face long after the original photo has technically been deleted.
Where regulators have drawn the line
None of this makes face-processing AI inherently harmful. These apps can be entertaining, they make sophisticated photo editing accessible to people with no design background, and face analysis more broadly supports things like account recovery, fraud prevention and identity verification. The FTC itself has acknowledged that biometric technology has legitimate uses. Its concern, and the concern of privacy regulators generally, is with companies that are not honest about how images are handled or how long biometric data sticks around.
The clearest cautionary tale remains the FTC’s case against Everalbum, the company behind a now-defunct photo storage app called Ever. The FTC alleged that Everalbum told users facial recognition would stay off unless they turned it on, while actually enabling it by default for almost everyone outside a handful of states and the European Union. The agency also alleged that Everalbum promised to delete the photos and videos of users who deactivated their accounts, then kept that content indefinitely, and used it to help build facial recognition technology it later marketed to enterprise customers, including some in law enforcement and the military. Everalbum settled with the FTC in 2021 and, under the terms of that settlement, was required to delete the retained photos and videos, delete the face embeddings it had derived from them, and destroy any facial recognition models or algorithms it had built using that data. It remains the FTC’s first enforcement action built specifically around the misuse of facial recognition technology.
Separately, the National Institute of Standards and Technology runs an ongoing federal testing program for facial recognition algorithms. Its landmark 2019 report on demographic effects found a wide range of accuracy across age, sex and race in the large majority of algorithms tested, a finding that later academic work has reinforced and that continues to inform how regulators think about deploying this technology for anything higher stakes than a filter.
What the law actually requires
In the European Union, the GDPR treats biometric data as a special category of personal information whenever it is processed to uniquely identify someone, which generally requires a stronger legal justification and tighter safeguards than ordinary personal data gets. In California, where I am currently located, the CCPA, as expanded by the CPRA, classifies biometric information used to identify a consumer as “sensitive personal information.” That gives Californians the right to know what is collected, request deletion or correction, opt out of having that information sold or shared, and limit how a business uses it.
Illinois has arguably the strongest biometric privacy law in USA. The Biometric Information Privacy Act, in place since 2008, requires a private company to give written notice of what biometric data it is collecting and why, and to get written consent before collecting it. Unlike most state privacy laws, BIPA lets individuals sue directly, with statutory damages of $1,000 for a negligent violation and $5,000 for an intentional or reckless one. Illinois lawmakers narrowed the law slightly in 2024, limiting a company’s exposure to one violation per person rather than one violation per scan, and confirming that an electronic signature satisfies the written consent requirement. For readers, the practical takeaway is that face-based AI is not an unregulated free-for-all everywhere. In some places, it triggers real legal obligations.
What to watch for before you tap upload
The clauses worth reading closely are the ones covering retention, reuse, model training, sharing with third parties, and deletion. A policy that says a company “may use your photos to improve our services” is often reserving more rights than a casual reader expects.
It is also worth checking whether an app wants access to your entire camera roll rather than a single photo, whether processing happens on your device or in the cloud, and whether the service links your face to a persistent account or device identity. The more a photo can be tied to a lasting profile, the higher the privacy stakes.
How to protect yourself
Treat these apps as something to use selectively rather than casually. Assume any uploaded face photo could be analysed for more than what shows up on screen, and avoid uploading sensitive or high value photos to services with vague retention language. Favor apps that clearly state whether images are deleted after processing and that let you opt out of having your photo used for model training. California residents can use their CCPA and CPRA rights to request access to, or deletion of, their data. Users in the EU and UK benefit from the GDPR’s stronger baseline protections around biometric data, and Illinois residents have BIPA’s written consent requirement working in their favor.
Conclusion
The AI photo craze is not only about vanity filters or a few seconds of amusement. It is also about how quickly a face can become a durable piece of digital infrastructure, one that outlives the app, the trend, and sometimes the user’s own memory of having uploaded it. None of this means people need to stop using these tools. It does mean understanding what is actually being handed over before tapping upload, because in the AI era, a picture is very often more than a picture.
(The author is a neuroscientist and Postdoctoral Fellow at the Salk Institute, California. His research focuses on Alzheimer’s disease, particularly the roles of iron, stress, and cell death. He also works to improve brain health access in underserved regions, including Jammu and Kashmir, through Teleprac Healthcare. Ideas are personal.)
Sources
- Federal Trade Commission, Policy Statement on Biometric Information and Section 5 of the FTC Act, May 18, 2023.
- Federal Trade Commission, press releases and complaint materials on the Everalbum, Inc. settlement, January 11, 2021 and May 2021.
- National Institute of Standards and Technology, Face Recognition Vendor Test (FRVT) program, including Part 3: Demographic Effects (NISTIR 8280, 2019) and the ongoing FRVT testing activity.
- University of Wisconsin-Madison News, “Popular social media apps use AI to analyse photos on your phone, introducing both bias and errors,” 2024, reporting on research presented at the IEEE Symposium on Security and Privacy.
- Meta, “New AI Tools to Help You Make Things Happen on Facebook,” about.fb.com, June 15, 2026.
- Google Photos Help, “Gemini features in Photos privacy hub.”
- European Union, General Data Protection Regulation, Article 9.
- California Civil Code Section 1798.140 and related provisions, as amended by the California Privacy Rights Act (CPRA).
- Illinois Biometric Information Privacy Act, 740 ILCS 14, and 2024 amendment (SB 2979).
