Sitting Down with Rajiv Shah

It has been a busy past few months, but I recently had the opportunity to chat with Rajiv Shah, an adjunct assistant professor in the Department of Communication at the University of Illinois at Chicago, and the author of the Smart Cameras blog. Below are some of my answers to his questions around privacy, vendor comparisons, industry connections to academia and the future of the smart camera market.

Again, very excited to share them with the growing IHF readership. Feel free to make comments on any of the responses or questions, and I will be sure to address them.

Also, check out Rajiv's Smart Cameras blog -- it focuses on developments in Chicago's use of video surveillance, as well as other "smart cameras" that utilize additional sensors and/or computer processing techniques. Cool read and a staple on my blogroll.

1. Privacy: What should the industry approach be towards privacy? Should they incorporate features that protect privacy? Should they have default settings that protect privacy or delete information? Or should we not worry about this? Is there a need for an industry-wide approach to this issue?

Conventional wisdom presents ‘privacy vs. security’ as a zero-sum game, one in which gains in one arena necessitate sacrifices in the other. And while there is certainly much truth in this, it is also equally true that in a modern society neither principle can exist without the other. There can be no security without privacy, and no privacy without security.

Today, any meaningful national security failure could create a response that curtails our civil liberties quite broadly. And conversely, next-generation security technologies deployed without adequate privacy and civil liberties protections likely face the crippling backlash of a concerned public. As a result, the security industry needs to take issues of privacy VERY seriously.

For me, an approach to privacy in the context of surveillance starts with a few key principles designed (1) to narrowly tailor a system’s use and (2) to ensure that system access is adequately controlled and audited. Today, conventional “dumb” surveillance systems offer none of these benefits. A large video wall in a security room or command center does nothing to distinguish between security threats and the average person; these systems cast an unnecessarily wide net, relying on human expertise and interest to filter down to focus in on actual security threats.

The issue of what to delete or keep in terms of surveillance becomes much less important on systems where this kind of ‘all-or-nothing’ approach to data access doesn’t exist. For instance, on a 3VR, an investigator might search through many months worth of video information looking for matches or clues relating to the kidnapping of a little girl. However, because this query is done algorithmically using facial recognition, and because the search request is logged and audited, there is ultimately much less concern about the overall retention of video data. The public generally has very little problem with legitimate surveillance investigation that doesn’t subject them to what they feel is needless voyeurism.

Retention of video also becomes less of a concern in the context of new blurring and encryption algorithms designed to protect individual privacy. These new technologies prevent generally tracking and identification of the pubic using surveillance, while preserving the ability of law enforcement and security officials to detect and investigate crime. To better understand what I mean by this, you should take a look at the recent article in New Scientist on some of what we are working on in 3VR labs right now.

In any case, issues of data retention, encryption, access control and the like are often more policy issues than industry issues. Instead, our focus should be enabling decision and policy makers to make, monitor and enforce these choices themselves. Our solutions should present options to do all of this...and more. Today, most security solutions don’t include any privacy protections whatsoever. That needs to change; asking someone to chose between security and privacy isn’t much choice at all.

2. Comparing Vendor Solutions: What can be done to make it simpler for end users to compare and contrast different solutions? It's very confusing now for end users to sort through claims by tens of companies on effectiveness, costs, technology, etc.

Normally, I would say that the answer to this question solely involves the emergence of various standards groups, independent testing and analysis organizations -- that is because the best response to confusion is nearly always more good information. And, I do think there is some good news on the horizon in both of those areas with new security analysts, bloggers and agencies entering the marketplace of ideas every day.

However, because many new solutions’ claims today are so specific and require real-world deployment for actual evaluation, the only way for end-users to fully educate themselves may be through pilot and testing projects that they conduct themselves. New technologies being offered today represent a quantum leap over previous generations of security and surveillance solutions, and end users will ultimately need to make a very significant investment in time and money to educate themselves on their benefits.

3. Connections to Academia: Explain if anything needs to be done to expand the connection between industry and academia. After all, much of the engineering talent has come directly from universities. Are there any suggestions you have for universities and their research?

The disconnect between commercial markets and academia is a classic problem seen across many industries, but I have noticed is a particular problem in the security industry. And as a partial result, there has been comparatively little innovation at the core of this market in recent decades. The surveillance methods used to catch criminals hasn't changed drastically with investigators still found staring at video walls or fast-forwarding through video stores looking for needles in haystacks. Plus, the innovations responsible for rapid productivity gains in knowledge workers in other industry segments seems to have largely passed this industry by. Who are security’s Googles, Microsofts, and Oracles?

To begin to address this issue, I think that most importantly security needs to become the province of innovative and interesting companies again. Only by tackling big, tough and important problems can the security industry hope to lure academia’s best and brightest, or focus them on its problems.

As for universities and their research, there is one problem faced by the security industry today greater than all others…and that is a crisis of our own making. It’s “information overload.” There are quite simply too many cameras and sensors today generating way too much information today, and the resulting torrent of data threatens to overrun our entire industry. Identify ways to process and sort and make meaningful this flood, and you will have done us all a great service…and there is probably a job waiting for you at 3VR, as well.

4. Future Growth of Smart Cameras: Have cameras hit a period of steady growth or do you foresee a potential boom ahead? If so, what are the crucial factors that you see that are limiting growth of that will cause growth to increase? Do we need to improve technology, better end-user experience, etc.

Cameras have seen explosive growth already -- sales worldwide are booming. Not only that, but the general sense of a 'camera' is evolving dramatically; dumb cameras, smart cameras, cameras that record at 200 frames per second, cameras integrated with iPods – they're popping up all over the place and exploding in ways that people could not have anticipated. Not only are the types of cameras available growing exponentially, but the data being collected by cameras has increased by a geometric factor far beyond that. New cameras have higher resolution, higher frame rates. More of just about everything!

As a result, we're stuck drinking from the firehose for the time being. We're inundated with data and have no idea what to do with it due to the sheer volume we're faced with. It's coming in too quickly to comprehend, and as a result, we've discovered that it's not the volume of data you collect, but what you can do with that video (and how quickly) that matters.

The modality of staring at a wall of video screens broadcasting camera streams broke down a long time ago – and we're better off for it. However, as camera volume, quality and speed explodes, we need to figure out how to comprehend and process this volume of data. If we're going to manage the growth of cameras, they don't need to be smarter – we're already capturing more data than we need – but rather more searchable and enable efficient retrieval of vital information.

CyberExtruder Gets 2D to 3D Face Patent

At the end of November, CyberExtruder announced that the company had been granted a new patent on their process for creating reliable 3D models of a person’s face from a single or series of 2D images.




Though the enforceability of this patent has yet to be tested, CyberExtruder’s 2D-to-3D conversion is certainly an important innovation to the security and biometrics technology industries. Why? It enables better matching between offline photographs and surveillance video. While the matching of "watch list" images against surveillance video has traditionally achieved mixed results as a result of inadequate lighting, angle, expression, etc., this patent could signify a leap forward in terms of the quality and value of 3D facial images.

In June of 2007, I wrote on how XID was using a similar technology in the “world’s largest” facial recognition access control project. In that instance, XID literally generated hundreds of thousands of variants of an enrolled employee’s face rather than using just a single 2D photo converted to a 3D model. Each day when an employee arrived for work, his or her photo would be taken and compared to the database of generated images rather than a single original. Interestingly, this approach generated huge improvements in the performance and accuracy of the Thailand access control system.



We’ll see if these two companies come into conflict over the new patent, but I don’t think they will. XID’s approach to 2D-to-3D is very different than CyberExtruder's-- almost quick and dirty by comparison. CyberExtruder, on the other hand, has become famous for its hyper-realistic…if sometimes creepy…generated floating 3D heads that lend themselves to applications well beyond security including gaming and movies, and even boast a fan in Phillip Rosedale of SecondLife.

Race Rec Face Rec

In most research and law enforcement circles, the prospect of using advanced facial recognition technology to determine an individual’s sex or race is an understandable gray area. Though census takers, demographers, and police reports have always concerned themselves with these details, the idea that a computer might reliably judge the racial background of each and every city passerby is more than a little creepy to some and raises the specter of racial profiling.

Recently Benjamin Wales, a graduating student at the Royale College of Art, set out to test peoples sensitivities on the subject by deploying his own "race detection" camera apperatuses [Spy Blog via Rajiv Shah] on the streets of London. And though he published no scientific data about the accuracy of his systems, it was clear that many who came across his art project shared Mr. Wales' mixed feelings on the technology.

Had scientific results actually been published in this project, however, I doubt they would have been terribly good. It turns out that this kind of image classification is awfully difficult to accomplish. To understand why, take a look at this research project at Mitsubishi Electric Research Laboratories (MERL).




In this research project, MERL succeeds in getting what you might consider "pretty good" results using two different image classification routines, Male vs. Female & Asian vs. Non-Asian, on excellent video footage of various faces. On thier own, these overly simple binary classifiers work well enough to justify further research, not well enough for any real world real-time law enforcement or profiling function.

Further, if MERL had added additional race classification outcomes, like black, or Korean, or Latino, etc., the level of accuracy they might be expected to achieve would decline considerably. And if they added enough racial classifications to approximate the actual diversity found in major cities, the algorithm would likely cease to provide any meaningful data at all.

One area where racial classification does show some promise, however, is in the the area of video search. The day is not very far away when a police official might be able to query a city surveillance system for an "Asian women with a red purse" when attempting to track down a suspected kidnaper and her victim. There would be a number of false matches, of course, and a human might still need to review a lot of video to ultimately close in on their intended suspect, but the search would be faster, more focused and sweep up fewer innocent bystanders if some face/person search algorithm was used.

I think this is a more realistic and reasonable use case for face classification...and one that might actually do some good, as well.

Too Much Information Makes Us Less Secure

How many cameras do you need to have before you can't see anything? How much video do you need to store before you can't find what you were looking for? Whatever that level is, the Brits passed it a long time ago when, as the author of this piece quips, the answer to the needle in the haystack problem became to "collect more haystacks."

"The answer in both America and Britain has been to collect more haystacks: useless, indiscriminately acquired information on people who've done nothing to arouse suspicion," he writes. "We even inveigle our citizens to become amateur curtain-twitchers and pecksniffs, demanding that they report "suspicious" activity to the authorities. Between DNA databases, mandatory fingerprinting for visa seekers, CCTV carpet-bombing, and Oyster card data, we've never collected more "security" information than we do today. But does this really make us secure? Is it possible to know too much?"

I think the question is not "Is it possible to know too much?", but rather "Is it possible for too much information to overwhelm our ability to know anything at all?". In which case the answer is a resounding yes! The delta between what security professionals can collect today and what they can process is called the "Security Gap" and it's getting bigger every day.

But, I'm not sure it's time to hoist the white flag just yet, however. A few years ago, after the bombing in London, closing the Security Gap meant assigning over one thousand MI-5 agents to manually review surveillance video. The process took over 6 weeks. Today, using new search and video analysis technology, just a handful of those agents could have probably completed much of the same work in a few days. Search technology in particular is rapidly closing the gap between the data we can collect and store and what we can "know". It's not unlike what happened with the Internet when tools like Google have made sifting through billions of images and pages as simple as typing a few key words. Suddenly this huge heap of information that had been sitting there unused by most became hugely useful when we could quickly and easily sift through it.

The prospect of these same advances in video search being applied to security is likely to greatly tame America and Britain's current jumble of haystacks, but such technology will of course bring with it a new set of problems. For those who have spent years working to close the Security Gap, I am reminded of another question about a dog chasing a firetruck.

"What's he going to do if he catches it?"

Popular Mechanic’s Panopticon

This month’s Popular Mechanics features a video camera on its cover and asks the question as to whether our “surveillance society” has gone too far.

James Vlahos, writes:

We have arrived at a unique moment in the history of surveillance. The price of both megapixels and gigabytes has plummeted, making it possible to collect a previously unimaginable quantity and quality of data. Advances in processing power and software, meanwhile, are beginning to allow computers to surmount the greatest limitation of traditional surveillance—the ability of eyeballs to effectively observe the activity on dozens of video screens simultaneously. Computers can't do all the work by themselves, but they can expand the capabilities of humans exponentially.

I think Vlahos does a pretty interesting job exploring both the benefits and pitfalls of some of our nation’s security initiatives and new technologies…and I am not just saying that because of his mention of my company, 3VR Security.

Here is what he had to say about us though:

Used by banks, hotels and retail stores, 3VR’s “searchable surveillance” systems automatically create a template of every face that passes in front of security cameras (it caught our author here at a Chicago hotel check-in counter). The system creates a mathematical model based on the geometry of each person’s face that can be compared to a central list of known suspects for instant alerts. The technology can also automatically log events based on an automated object recognition analysis of an entire scene—for example, Frank Jones met with Doris Meeker at 12:45 pm; Meeker arrived in a blue sedan. Because all events are cataloged, several months’ worth of data can be analyzed in minutes.

One point Vlahos doesn’t make in his article, however, relates to the dual-benefit of many of these new surveillance technologies. While traditional security approaches are not particularly effective or conducive to privacy, new more effective technologies don’t necessary bring with them even greater privacy issues. For instance, the use of search engines and video analysis greatly increases the chance of catching bad guys before they strike. But, these same technologies can be used to help audit surveillance monitoring efforts and generally limit their abuse, as well.

The Era of Searchable Surveillance

From my article in Security World Magazine:

At present, security systems are inefficient, lack integration and are not
scalable to meet the needs of large or geographically dispersed
organizations. The challenge is heightened by the rapid convergence of
physical and IT security systems. Existing reactive alarm systems and raw
video review are insufficient for overcoming the wide and diverse range of
threats facing companies and government organizations, leaving them vulnerable
to security breaches.

Real Life James Bond

If you are into spy gadgets and other 007 goodness, look no further than the "CIA's Venture Fund" In-Q-Tel. The name is actually derived from the combination of the word "intel" and the character "Q" from the Bond books.

Since In-Q-Tel was founded in 1999, the firm has reviewed more than 6,300
business plans for everything from identity recognition software to nano-sized
electronic circuits. Many proposals come in via its Web site. In-Q-Tel has put
about $200 million into more than 100 companies, beating traditional VC
investors to technologies such as the mapping software that's become Google
Earth.

One of their investments is a particular favorite of mine.

Another part of In-Q-Tel's video strategy is the 3VR investment with Kleiner Perkins. After the London subway bombings on July 7, 2005, about 1,000 British police spent six weeks sifting through video from 6,000 surveillance cameras, says Russell, who founded the company in his basement in 2003....With 3VR, every time someone passes in front of a camera, the software assigns an identification number and establishes a profile based on the geometry of the person's face. When the face is captured from a different angle or in different light, the software creates another mathematical model. The system can be programmed to recognize faces and to issue alerts once a suspect is spotted.

Bad guys beware.

Search Engines Using Facial Recognition

Search engines adopting face recognition now include Google, Microsoft Live, Exalead.
Blog, Search Engines are getting better as everyday passes by, and we see new features added to give users the best experience and satisfaction ever. Face Recognition is a technology that has been adapted by search engines like Google, Live Search and Exalead in a move to tune up their image search results. Now you could restrict image searches to only human faces, rather than getting more of irrelevant images. (Via Nitesh Gautam’s Blog)

Reuters launches facial video search web site using Viewdle. Not bad.