Introducing Flock’s AI-Powered Search Solution for Law Enforcement

Flock informs WIRED that law enforcement cannot conduct searches using prohibited characteristics like religion and nationality, asserting that any attempt to search using these terms “will be blocked.” When inquiring about the circumstances that trigger warnings in these categories, the company did not provide an answer.

If a T-shirt or bumper sticker triggers a warning due to text with “constitutional protections,” officers are notified that their search will be logged and that administrators will be alerted. They are then required to check an acknowledgment box and provide a comment before they can proceed by clicking “Continue With Search Anyway.”

Flock maintains that the warning allows for legitimate investigations, citing the example of a victim describing a suspect wearing a biker gang jacket with “a certain gang logo or emblem containing a flag or other insignia.” If an officer continues the search, they must report to an administrator within their department for evaluation.

Kate Ruane, who directs the free expression project at the Center for Democracy and Technology, has spent years examining automated moderation systems. She notes that political, social, and cultural expression is “an incredibly amorphous category,” and one reason an officer might include such language in a search is to identify participants in a protest. This has already occurred. The category is concerning enough for Flock that it issues a warning, but she emphasizes, “you can still obtain the results if you just click through.”

“No large-scale content moderation is likely to be completely accurate,” she states. She adds that analyzing moving video is even more challenging and fails more frequently.

Ruane mentions that the categories which are blocked can be circumvented. For instance, while the system prohibits searches by religion, it does not restrict searches by attire, allowing officers to potentially evade the block by searching for distinctive clothing associated with a specific faith. “Many individuals may have their images returned from these types of queries and would likely be upset if they were aware of it.”

Last year, a California officer searched for “American flag.” The search was blocked when it targeted a person but uncovered 11,000 cameras when directed at vehicles instead.

A warning primarily deters those officers who are not already inclined to proceed with the search, according to Deepak Kumar, an assistant professor of computer science and engineering at the University of California San Diego. Kumar, who researches trust and safety systems, highlights examples of interfaces designed to combat online harassment, where motivated users sometimes cause more damage after receiving a warning, as well as browser alerts aimed at guiding users away from malicious websites, where the effectiveness hinges on how warnings are presented.

“This type of logging can be beneficial for administration, allowing an examination of which officers or end-users frequently bypass warnings, but its value is contingent on whether it is properly managed,” Kumar says. “It serves less as a deterrent and more as a record.”

Scrutinizing search terms before a query executes will certainly catch various instances of misuse, but alone, such precautions are known to falter without equal focus on the responses generated by the model. “Best practice involves evaluating both inputs and outputs,” Kumar insists, “which is why many AI safety initiatives now assess both to prevent harmful outputs.”