Face Swapping 2.0 - Now Mostly Prettier

In July, I made a post regarding new technology from Columbia University that swapped faces in photos and videos with generic amalgems drived from a photo database. The technique, which had potential privacy benefits...and even certain Hollywood applications...suffered from one major drawback. The modified faces were really really really ugly.

See below (From Columbia University Research Paper):



Tough to stomach, I know, but these results were bound to be approved someday, and thankfully that day is here. The New York Times is reporting that computer scientists in Israel have taken a more beauty-centric approach to face swapping that simply replaces faces in images and video with more idealized representations of the same faces. Their software applies an algorithm covering over 200 different facial measurements that brings a target face closer to the ideal. In most cases, this approach goes well beyond simple airbrushing, dramatically altering a person's appearance and perceived identity.

See below (From Lars Klove for the NYT):



I am not sure either technology is perfectly suitable for privacy protection in video surveillance just yet, but given the choice between the two approaches above, for aesthetic reasons, I'd go with the latter.

Amazing Video Enhancement Technology

Researchers at the University of Washington recently released a video showing a series jaw-drapping video enhancements now possible using various image analytic and modeling techniques. In one demo of interest, simply by mixing a few high-resolution photographs of a scene with a lower quality video stream, the scientists were able to dramatically enhance the quality and resolution of the complete video...by 4x.

The technology could also be used to seemlessly remove private content from surveillance video, truely making proctected images of people or objects invisible. That's probably welcome news to some, but disconcerting to law enforcement professials who already have significant concerns about the reliablity of photos presented them. Video has been generally thought of as much harder to manipulate...no more.

Face Swapping Still Not Pretty

Not since John Travolta and Nicolas Cage swapped mugs in the 1997 non-hit Face/Off, have I been so intrigued/disturbed by the prospect of changing faces. Researchers at Columbia University have developed a system capable of quickly replacing and/or obscuring faces in photographs with new composite or synthetically generated images. Unlike our technology at 3VR and that at Google, which only blurs faces, this technology actually changes them...with mixed results.

Here is an example:

The authors see several applications to this technology from protecting privacy in public images to Hollywood special effects. But, as you can see above, the researchers' favorite application seems to focus on making famous faces less attractive.

Too Much Information Makes Us Less Secure

How many cameras do you need to have before you can't see anything? How much video do you need to store before you can't find what you were looking for? Whatever that level is, the Brits passed it a long time ago when, as the author of this piece quips, the answer to the needle in the haystack problem became to "collect more haystacks."

"The answer in both America and Britain has been to collect more haystacks: useless, indiscriminately acquired information on people who've done nothing to arouse suspicion," he writes. "We even inveigle our citizens to become amateur curtain-twitchers and pecksniffs, demanding that they report "suspicious" activity to the authorities. Between DNA databases, mandatory fingerprinting for visa seekers, CCTV carpet-bombing, and Oyster card data, we've never collected more "security" information than we do today. But does this really make us secure? Is it possible to know too much?"

I think the question is not "Is it possible to know too much?", but rather "Is it possible for too much information to overwhelm our ability to know anything at all?". In which case the answer is a resounding yes! The delta between what security professionals can collect today and what they can process is called the "Security Gap" and it's getting bigger every day.

But, I'm not sure it's time to hoist the white flag just yet, however. A few years ago, after the bombing in London, closing the Security Gap meant assigning over one thousand MI-5 agents to manually review surveillance video. The process took over 6 weeks. Today, using new search and video analysis technology, just a handful of those agents could have probably completed much of the same work in a few days. Search technology in particular is rapidly closing the gap between the data we can collect and store and what we can "know". It's not unlike what happened with the Internet when tools like Google have made sifting through billions of images and pages as simple as typing a few key words. Suddenly this huge heap of information that had been sitting there unused by most became hugely useful when we could quickly and easily sift through it.

The prospect of these same advances in video search being applied to security is likely to greatly tame America and Britain's current jumble of haystacks, but such technology will of course bring with it a new set of problems. For those who have spent years working to close the Security Gap, I am reminded of another question about a dog chasing a firetruck.

"What's he going to do if he catches it?"

Surveillance as Handy Marketing Tool

A lot of new products are coming out that help companies use existing CCTV surveillance technology for marketing purposes--capturing everything from basic data about how many people stopped at a promotional display to more advanced details about particular customers. In addition to the camera-and-box equipped billboards being piloted by TruMedia and Quividi, Google recently announced its partnership with a company called Xuuk to produce a palm-sized camera called the Eyebox that will track how many times people look at both billboards and products in stores. The idea is to provide brick-and-mortar stores or companies the same tracking abilities in real life as they have with Google ads online. Personally, I think using face recognition with an already existent system (like, I don't know, a 3VR system!) makes better financial sense than spending $25,000 for a separate system and cameras, but even above and beyond that, using surveillance systems as marketing tools doesn't exactly help to assuage the public's "Big Brother" concerns about surveillance.

Finally, A Balanced Take on Security Cameras

Okay, in this case, Fox News actually is providing a "fair and balanced" take on things. Apparently security camera neither put a complete stop to crime nor are completely useless privacy invasions. In known, high-crime areas, the deployment and use of surveillance systems can actually help fight crime. Maybe not all crime, but at least enough to make it worth doing. Obviously, privacy is important. I get that. But I think the way to get folks on board with privacy protection is not to make unfounded statements like "cameras have absolutely zero effect on crime." That's just absurd and patently false. Extreme statements like that will just keep people on opposite sides, not solving either the privacy problem OR the security problem.

On a lighter note, with the suggested increase of cameras in Lynwood, Paris Hilton's crime spree may finally come to a halt.

More Cool Privacy Tech

The new privacy technology we are working on at 3VR does more than simply blur people's faces. Here is an example where full images of surveilled people are actually scrambled and encrypted before they are displayed for monitoring security personnel.

The image on the left is from the original video feed. The second image is from a scrambled and encrypted version of that same feed.

Though with the encrypted feed it's possible to detect loitering, fighting, and many other behaviors of concern, identity information is totally protected...that is, until an authorized user chooses to decrypt the feed. BUT then that action is controlled and logged, and an alert can even be generated, to ensure that the surveillance system is not being abused or misused.

The Costs and Benefits of Face Recognition

Most people think face recognition is great when it's catching baddies and weird or invasive when it's tracking innocent people. The technology has sparked debate after debate over whether privacy or security is more important. Honestly? I think we can have both.

First off, people need to understand what facial recognition can and can't do. This ABC story about a face rec system analyzing someone's face and telling a clerk that they're underage is bogus. If the kid had been in the store previously and been busted for buying liquor then yes, the system could alert the clerk, but no one is suggesting that face rec can tell the difference between 17 and 18 any more than a human can.

One thing face rec can do, which hasn't gotten a ton of press, is include simple privacy measures. We have been working on this at 3VR and I wrote recently about a team in Canada that's working on a similar project. Basically, software engineers can write a password-protected program that blurs faces and when an incident occurs, an investigator can unblur faces in particular pieces of video. This way, while people may still feel uncomfortable about being on camera, at least they will not really be watched unless they happen to be present during a robbery or some other incident, in which case they'll typically be glad the cameras were there to help catch the bad guys. Also, because this type of application can also have auditing capabilities written into it, it provides a crucial and often overlooked capability: a way to "watch the watchers," if you will.

The same could feasibly work in the grocery store situation described in the ABC story - if someone was a match with a suspect in the database, then the system could alert the clerk. For everyone else, faces could be blurred, and if someone is caught buying liquor or cigarettes underage, then the store manager could unblur the face and save it to the suspect list.

There has also been a lot of press recently about the rise of surveillance that risks privacy without actually improving security. I agree. Thing is, the bulk of new cameras installed are meant to catch traffic violators and raise money for municipal governments, not improve security. These cameras misfire fairly often, sometimes costing a city more than they're worth, and invading citizens' privacy for no good reason.

During the bombings in London, however, investigators were able to use video footage to find their suspect. If they had had face recognition and video search capabilities, that investigation would have been far shorter. And as banks have begun installing surveillance systems, they have seen a marked increase in the number of fraud cases they're able to solve.

Surveillance clearly has a place in modern society, but I do think that the industry needs to continue to work towards securing both people and their privacy.

Should Red Light Cameras Be Stopped?

Theoretically red light cameras are great - who wants people running red lights? The problem comes when people fail to differentiate between red light cameras and security cameras and wind up throwing the baby out with the bath water. While they appear to be associated with safety, red light cameras in fact exist to generate revenue for cities, and they're largely ineffective at actually reducing accidents. Security cameras, on the other hand, exist to protect cities and people. To lose the latter because of the annoyance and poor performance of the former would be silly in the best case scenario, tragic in the worst. Unfortunately, that is exactly what may happen in cities throughout the country as citizens protest against red light cameras and the trade-off of privacy for city funds that they require.

Privacy-Enhancing Video Technology

It's great to see people trying to address the privacy vs. security question with new solutions instead of complaints on either side. Today, video surveillance is not particularly private nor effective, and we really need both. A new technology developed by researchers at the University of Toronto adds a sort of face blur filter to security video that can be reversed with a decryption key. So, the faces of the innocent remain fuzzy and out-of-focus, while investigators zero in on suspects.

The University of Toronto tech was most likely at least partially motivated by changes in Canadian privacy laws. Even without such laws, technology like this is important and certainly helps to address the privacy issue. At 3VR we are currently working on our own privacy enhancing technologies that also incorporate advanced facial recognition techniques. The technology helps to hide the faces of the innocent while at the same time identifying bad guys, providing a solution that gives equal weight to privacy and security, which is something I think should be at the center of any city surveillance deployment.

Rethinking Surveillance

Video surveillance has become a fact of everyday life. Each time you withdraw cash from the corner ATM, travel through an airport or visit a national monument, your image is probably being recorded. But you may be surprised to learn that there are no federal laws governing how these images can be used, where they should be stored, with whom they may be shared and when they must be destroyed. In this age of YouTube, TMZ and "Cops," it's hard to know where your image might reappear.

The laissez-faire approach of our national legislators is no longer an option. As an increasingly sophisticated surveillance blanket covers more of the United States, we need federal laws to preserve an individual's right to privacy while setting principles governing the use of closed circuit television and other surveillance technologies for bona fide security purposes.

Surveillance technologies will continue to gain in capability -- and become more intrusive. Issues of privacy and public surveillance may appear vexing, but the United States must move forward with laws to effectively adapt to the inevitable spread of this technology. If the public is to trust business and government to watch over us, we need to follow the lessons of Britain and protect video images as we do other private data.

FBI Wants Palm Prints, Eye Scans, Tattoo Mapping

CLARKSBURG, West Virginia (CNN) -- The FBI is gearing up to create a massive computer database of people's physical characteristics, all part of an effort the bureau says to better identify criminals and terrorists. But it's an issue that raises major privacy concerns -- what one civil liberties expert says should concern all Americans. The bureau is expected to announce in coming days the awarding of a $1 billion, 10 year contract to help create the database that will compile an array of biometric information -- from palm prints to eye scans.

Given the recent request for a $3 trillion budget, $1 billion doesn’t seem like all that much anymore.

Cam Case Collateral

The rapid proliferation of video cameras in the public, and their increasing importance in court room had brought forth a number of important ethical and legal questions for those “caught up in the net” of video surveillance.

Among the questions to be answered: can the cops resist the temptation to "go fishing" for other bad guys when reviewing tape of an incident? What are the ethical ramifications of overlooking minor infractions caught on tape and, most importantly, will the prospect of being filmed actually dissuade ne'er-do-wells from ne'er-do welling?

While we will never be able to completely eliminate these issues of “collateral damage” in surveillance-centric criminal investigations, we can do quite a bit with recent technology advances to minimize them. For instance, the days of needing to troll through days and days of video to track down a particular bad guy are over. Using searchable surveillance and biometric technologies, an investigator can target a video investigation on a particular individual. The video search engine returns only that video that relates to the subject in question.

Sometimes, modern surveillance technologies are viewed exclusively through the lens of ‘big brother,’ but in this case we can also see how those same technologies can be used to protect our privacy by bringing much needed focus and efficiency to video investigations.

Australian Government Revs up LPR

CrimTrac, the Australian government's criminal information and intelligence agency, is taking the first steps toward developing a nationwide automatic number plate recognition system to capture the vehicle details of suspects and citizens alike.

"We looked at the U.K. experience as part of the study; it's probably the most advanced integration of ANPR technology anywhere in the world, and we'll draw on those experiences, but primarily it will have to suit the Australian context," said Darren Booy, ANPR Project Manager for CrimTrac.

Australia is studying its Automatic Number Plate Recognition (ANPR) system to determine the feasibility of implementing a national network. If approved the rollout should take one to four years. With regards to civil liberties, the study will also include a comprehensive privacy impact assessment after widespread use of the technology overseas raised the concerns of privacy
advocates.

Facebook: The New Look of Surveillance

http://www.alternet.org/story/72556/

Facebook sparked an immediate national uproar when it was launched in 2002. Today, the activism has waned, and the surveillance continues largely unabated.

People know their actions are tracked online, says a Facebook spokesperson, just
as they're tracked on streets filled with surveillance cameras, whether
privately controlled through an ATM or publicly controlled [for] legitimate
anticrime or anti-terrorism purposes. In an era of massive top-down
surveillance, posting information on a website may feel downright redundant.

Facebook's growing dominance reflects a society that is increasingly complacent with spying. And while social networking is a free and convenient service, abdicating control of personal information, photos, writing, videos, and memories seems like a high price to pay

Axis of Evil?

Privacy International and Electronic Privacy Information Center have named the U.S., U.K., China, and Russia as the four worst "endemic surveillance societies."

Here is what the report had to say about China:

• Limited rights under constitution under articles 37, 38, 39
• Chinese government acknowledges that it has room for improvement in applying laws fairly and systematically
• Stricter controls are being exerted on press, internet, academics, lawyers and NGO's
• Extensive surveillance schemes implemented in anticipation of the 2008 Olympics
• Increased expectation of privacy amongst citizens has led to academics calling openly for stronger privacy laws
• Some privacy laws
• Search and interception does require warrants but they are authorised by officials and prosecutors
• Increased legal activity and suits in the area of medical privacy
• In 2006 China's central bank developed a database that links up information on consumer credit; and private sector initiatives are emerging that advertise access to 90 million incomes, marital status and sensitive information for 12 cents per request
  

Here is what it had to say about the United States:

• No right to privacy in constitution, though search and seizure protections exist in 4th Amendment; case law on government searches has considered new technology
• No comprehensive privacy law, many sectoral laws; though tort of privacy
• FTC continues to give inadequate attention to privacy issues, though issued self-regulating privacy guidelines on advertising in 2007
• State-level data breach legislation has proven to be useful in identifying faults in security
  REAL-ID and biometric identification programs continue to spread without adequate oversight, research, and funding structures
• Extensive data-sharing programs across federal government and with private sector
• Spreading use of CCTV
• Congress approved presidential program of spying on foreign communications over U.S. networks, e.g. Gmail, Hotmail, etc.; and now considering immunity for telephone companies, while government claims secrecy, thus barring any legal action
• No data retention law as yet, but equally no data protection law
• World leading in border surveillance, mandating trans-border data flows
• Weak protections of financial and medical privacy; plans spread for 'rings of steel' around cities to monitor movements of individuals
• Democratic safeguards tend to be strong but new Congress and political dynamics show that immigration and terrorism continue to leave politicians scared and without principle
• Lack of action on data breach legislation on the federal level while REAL-ID is still compelled upon states has shown that states can make informed decisions
• Recent news regarding FBI biometric database raises particular concerns as this could lead to the largest database of biometrics around the world that is not protected by strong privacy law

Notice anything odd? One would think that the U.S. afforded far fewer privacy protections to its citizens than China! No serious person could make that argument. And the report is completely silent on whole swaths of the planet where public policing of moral behavior is the norm and people are regularly put to death for their “crimes” in the bedroom. As someone who takes privacy and personal liberty pretty seriously, I found Privacy International’s report to be more than a little frustrating. Every time privacy advocates verge from objective advocacy into meaningless bashing of Western institutions, the genuine cause of privacy is set back and the thoughtful balancing of the worlds privacy and security concerns is made that much more difficult.

Look!

The new trailer is out:

Sex on the Beach

In addition to sand flies and the occasional tsunami, devotees of beach-front-fornication in Palm Beach have something else to worry about… talking surveillance cameras.

Europeans Like to Be Watched

Unisys recently released results of a study of European attitudes toward security and privacy :

The Unisys Security Index is the first of an ongoing global research project to help businesses and governments understand consumer attitudes to national, personal, financial and Internet security. Assessing the opinion of more than 13000 people in 14 countries, across Europe, the US, Brazil and Asia-Pacific, the survey will be repeated three times a year.

Among the key findings:

Widespread acceptance of new security measures including video surveillance and increased security when travelling. On the Continent, four in five people are comfortable with an increase in video surveillance and increased security when travelling - especially at airports.

FBI Has It's Ways

George Christian, Executive Director of a consortium of 27 libraries, received a National Security Letter, in July of 2005 after a library patron’s online activity garnered the attention of the FBI. As a result, until recently he’s been under a strict gag order:

Christian is just one of an estimated 300,000 people who have received National
Security Letters requesting access to personal data. Phone records, Internet
usage, bank statements and telephone conversations are just some types of
information agents have gathered either through National Security Letters or
surveillance.

Interesting. And then there is this tidbit:

“I said you are asking for what we know about the user of an IP address [a
series of numbers that identifies a specific user on the Internet] for 45
minutes five months ago,” Christian said. “There’s just no way. And quite
seriously the agent looked at me said, ‘No, we have ways.’ ”

Warrantless Surveillance Down Under

Via ABC News Australia:

Premier Morris Iemma says the new laws will give police greater power to
install, monitor and retrieve a range of surveillance devices. […] Under the
changes, police will be able to use surveillance devices without a warrant if
there is an imminent threat of serious violence to a person or substantial
damage to property.

Not So Fast Ontario

The $21 million Toronto Transit System is on hold after Privacy International filed a complaint with the city’s privacy commissioner. They believe “that the installation of cameras on the scale proposed by the TTC fundamentally violates privacy law.”

The TTC, which provides 1.4 million rides each weekday, is in the process of installing up to 10,000 security cameras in its buses, streetcars and subway system, adding to its current network of about 1,500 cameras. The system, which was approved by the TTC last spring and is expected to be operational by June, will be capable of snapping photos and recording video – and in some cases, audio – of any of the TTC's daily riders. The federal government kicked in $6.5 million for the project.

Like all previous attempts to slow and or block city camera implementations, this effort is probably doomed to failure.

TTC chairman Adam Giambrone defended the system today, saying the information is centrally collected and accessible only to police, and that the cameras are part of a larger security plan that involves such measures as increasing the number of transit constables. "We were the last of the major transit authorities in North America and Europe – who are way ahead – to install a major camera program," he said. "So clearly, the consensus out there is that this is a positive."

Given their track record and the inertia behind these urban camera rollouts, privacy advocates might want to try a new tact. Maybe…something like this or this.

Chicago to Add Cameras to Street Sweepers

For those of you who were worried about privacy implications of “Google’s Street View,” Chicago is about to do them one better:

“We already know we're under surveillance ... well, pretty much all the
time. But it looks like the City may be adding some more robot overlords in
the near future: City Hall is looking for companies to provide high-res
cameras to be attached to street sweepers to photograph illegally parked
cars. The cameras would capture a pic of your license plate, and you'd get a
ticket in the mail. Most street sweeping tickets are $50.”

I am not sure MORE parking tickets were what folks had in mind when they stood up for more city surveillance.

Surveillance...In Your Face

Pretty funny, but it’s not exactly the kind of serious debate we really need right now when it comes to managing issues of security and privacy.

Six Minutes to Midnight

Today, the ACLU released its report “Even Bigger, Even Weaker: The Emerging Surveillance Society,” and with it unveiled a Surveillance Society Clock on its website – set at six minutes to midnight. While a great attention-grabbing device, the ACLU’s privacy doomsday clock and report warn of the dangers of technology without showing the other side of the argument – that new advances in surveillance technologies can also be used to strengthen privacy. At least, compared to the current situation – in which surveillance is not particularly effective at either catching bad guys or protecting civil liberties.

Take CCTV and video surveillance. Today, 30 million cameras in the United States create 4 billion hours of video per week, and the number of cameras is expected to more than double over the next few years, according to market research and consulting firm J.P. Freeman.
Yet until now, the only way to make use of the video generated has been for people to manually review it. Video surveillance helped identify the perpetrators of the London Underground suicide bombings of July 2005 – but police officers had to spend thousands of hours looking at CCTV footage to find the relevant information. Manual review of video evidence is inefficient and expensive. Moreover, there’s no good way to track what the viewer is actually looking at. But new advances in video search, storage, and analytics not only make it easier to find important information, they also make it easy to keep an auditable record of the type of footage the reviewer searched for and replayed.

Human nature is what it is. Yet knowing that there’s now an accessible record of what someone has looked at (or emailed or downloaded) could make people think twice – about reviewing non-relevant titillating CCTV footage again and again, rather than checking for suspects to a crime that’s occurred. Or sending suggestive emails to teenage congressional aides, or downloading pornography at work.

The ACLU’s concerns about privacy and civil liberties are not unjustified. But as with just about all new technologies, tools for better surveillance can be used for good and ill.
Evidentiary DNA testing and DNA databases generated controversy when they were first proposed. Yet without them, the Innocence Project and its affiliates could not have gotten more than 200 people – many of them on death row – exonerated from the serious charges they were previously convicted of. The Innocence Project has been doing this since 1992, only a few years after DNA evidence was first used in the legal system, after being created as a pro-bono clinic by DNA legal experts Barry Scheck and Peter Neufeld of the Benjamin R. Cardozo law school in New York. Clearly, this is a case where people saw how a new technology with significant potential effects on privacy could also be very effectively used to restore and strengthen civil liberties.

Another example is the ability to retrieve people’s online search records. That too led to controversy about loss of privacy – but when properly used with a warrant to identify murder suspects like Robert Petrick (shown to have googled “neck snap break” shortly before the death of his wife) or child pornographers, people clearly approve of the technology’s social worth.
And Home Electronic Monitoring Systems (a.k.a. ankle monitors) to maintain house arrest conditions were called Orwellian when they were first introduced in the 1980s. Since then, they’ve effectively worked to keep non-violent criminals off the streets and out of overcrowded prisons.Instead of merely decrying them as privacy-killers, those who fear new advances in surveillance technology that make more efficient tracking of individuals’ activities possible should take a good hard look at their context.

In a more enlightened era, privacy advocates and surveillance-tool users and creators should all be strongly encouraged to look at the context and history of similar advances. Instead of shouting at each other from opposite sides of the fence, or ignoring the other side, we should initiate forums for true discussion about these tools. There, we should take into thorough consideration both the effects on privacy and the societally valuable uses of the technologies for security and law-enforcement. Finally, ensuring that the use of these surveillance tools is auditable, both technologically and by law, is crucial.

If we commit to bringing more balanced views to the debate, carefully hearing both sides of the privacy vs. security argument, we can avoid the doomsday clock scenario. With better analysis of surveillance information and strong, automated auditing of that analysis, we can have both better surveillance, to help catch and foil people who are doing genuinely bad things, as well as strong civil liberties and privacy.

I agree with the ACLU that the U.S.’s spotty, patchwork laws around privacy need to be of the privacy vs. security argument, we can avoid the doomsday clock scenario. With better analysis of surveillance information and strong, automated auditing of that analysis, we can have both better surveillance, to help catch and foil people who are doing genuinely bad things, as well as strong civil liberties and privacy. overhauled, made consistent, and stronger – so that we at least catch up with just about every other developed country. And when we do, the new technologies the ACLU cites as cause for concern are also what could save the day in helping protect privacy.

ACLU Surveys California Surveillance

The ACLU issues a report today entitled “Under the Watchful Eye.” While not terribly thoughtful in their analysis, they do manage a few interesting data points on California’s surveillance infrastructure:

The ACLU conducted a public records survey of 131 jurisdictions throughout the
state. Among the key findings:

· 37 cities have some type of video
surveillance program
· 18 cities have significant video surveillance
programs of public streets and plazas; an additional 10 jurisdictions are
actively considering such expansive programs
· 18 cities have systems in
which police actively monitor the cameras