virtual contact sheets - cs.cmu.eduworichte/talks/pdlretreat... · a contact sheet approach to...

Virtual Contact Sheets Wolfgang Richter

Kiryong Ha, Alok Shankar, Ardalan Amiri Sani*, Jan Harkes, Lin Zhong*, Mahadev Satyanarayanan

PARALLEL DATA LABORATORY Carnegie Mellon University

*Rice University

Crowdsourcing the News [1] “When every cell phone can record

video and take pictures, everyone is a potential news source.”

Wolfgang Richter © November 11 http://www.pdl.cmu.edu/ 2

Who killed Ian Tomlinson?

CNN: Caught Red Handed [2]

Wolfgang Richter © November 11 http://www.pdl.cmu.edu/ 3

I’m a sneaky thief.

Wolfgang Richter © November 11 http://www.pdl.cmu.edu/ 4

Problem

Cloud

Recent, not uploaded

Older, uploaded

Fundamental Mobile Challenges • Real-time sync not possible:

• Energy •  Infinite battery life desired

• Bandwidth •  Increasingly metered and charged

Wolfgang Richter © November 11 http://www.pdl.cmu.edu/ 5

How far can we degrade the objects in a search before it loses meaning?

Just-in-Time Search 1.  Check virtual contact sheet

•  Collection of fidelity-reduced images in the cloud

2.  Pull full fidelity image on hit •  Search full fidelity; present full fidelity results

to searcher

Wolfgang Richter © November 11 http://www.pdl.cmu.edu/ 6

Cloud

Wolfgang Richter © November 11 http://www.pdl.cmu.edu/ 7

Near Real-Time Sync (1)   Search hit (2)   Retrieve

Contact Sheet Analogy

Wolfgang Richter © November 11 http://www.pdl.cmu.edu/ 8

Evaluating Feasibility [3]

Wolfgang Richter © November 11 http://www.pdl.cmu.edu/ 9

Face Detection, RGB Histogram, Brightness Filter

Benchmark

Using Varying Resolution and JPEG Quality Factor

Energy Savings

Wolfgang Richter © November 11 http://www.pdl.cmu.edu/ 10

0

1000

2000

3000

4000

5000

6000

7000

Full 640x480 320x240 160x120 80x60 40x30 20x20

En

erg

y C

on

sum

ptio

n (

mJ)

Fidelity

Energy

10 images = 5 seconds call time (Droid)

Bandwidth Savings

Wolfgang Richter © November 11 http://www.pdl.cmu.edu/ 11

0

50

100

150

200

250

300

350

400

Full 640x480 320x240 160x120 80x60 40x30 20x20

Bandw

idth

(K

B)

Fidelity

Bandwidth

Orders of magnitude savings in bandwidth.

What about Search Accuracy?

Wolfgang Richter © November 11 http://www.pdl.cmu.edu/ 12

0

0.2

0.4

0.6

0.8

1

640x480 320x240 160x120 80x60

Filt

er

Acc

ura

cy

Fidelity

0.650

0.424

10095755025

No one-size-fits-all sweet spot.

Face Detection

Conclusion & Future Work •  Searches retain meaning

•  No one-size-fits-all, but •  Orders of magnitude bandwidth savings •  Significant energy savings

•  More search primitives? •  Perceptual hashing •  Face recognition •  Texture Filters

•  Better fidelity reduction? •  Keep faces at high fidelity

Wolfgang Richter © November 11 http://www.pdl.cmu.edu/ 13

Further Reading [1] TED. http://www.ted.com/talks/paul_lewis_crowdsourcing_the_news.html.

[2] CNN. http://www.cnn.com/2010/CRIME/08/24/new.jersey.theft.photo/index.html?hpt=C1.

[3] SimpleCV. http://simplecv.org/.

Jan Harkes, Mahadev Satyanarayanan, Ardalan Amiri Sani, and Benjamin Gilbert. A Contact Sheet Approach to Searching Untagged Images on Smartphones. CMU Technical Report CMU-CS-11-132.

Ardalan Amiri Sani, Wolfgang Richter, Xuan Bao, Trevor Narayan, Mahadev Satyanarayanan, Lin Zhong, and Romit Roy Choudhury. Opportunistic Content Search of Smartphones. Rice University Technical Report TR0627-2011.

Xuan Bao, Trevor Narayan, Ardalan Amiri Sani, Wolfgang Richter, Romit Roy Choudhury, Lin Zhong, and Mahadev Satyanarayanan. The Case for Context-Aware Compression. HotMobile 2011.

Jason Flinn and Mahadev Satyanarayanan. Energy-aware Adaptation for Mobile Applications. SOSP 1999.

Y. Nimmagadda, K. Kumar, and Y. H. Lu. Energy-efficient image compression in mobile devices for wireless transmission. ICME 2009.

Wolfgang Richter © November 11 http://www.pdl.cmu.edu/ 14

Full Fidelity

Wolfgang Richter © November 11 http://www.pdl.cmu.edu/ 15 12,000+ Flickr dataset

Average photo size: 374 KB

Reduce Resolution

Wolfgang Richter © November 11 http://www.pdl.cmu.edu/ 16 12,000+ Flickr dataset

Average photo size: 221 KB

JPEG Quality Factor

Wolfgang Richter © November 11 http://www.pdl.cmu.edu/ 17 12,000+ Flickr dataset

Average photo size: 6 KB

Bandwidth Savings Table

Wolfgang Richter © November 11 http://www.pdl.cmu.edu/ 18

Resolution 100 95 75 50 25 5

640x480 221 KB 114 KB 47 KB 31 KB 19 KB 6 KB

320x240 61 KB 33 KB 14 KB 10 KB 6 KB 2 KB

160x120 17 KB 10 KB 5 KB 3 KB 2 KB 826 B

80x60 5 KB 3 KB 2 KB 1 KB 892 B 460 B

40x30 2 KB 1 KB 725 B 588 B 484 B 345 B

20x20 1000 B 727 B 501 B 436 B 385 B 316 B