Picky Pixels

Nobel Pixels

2009-10-23T10:46:00.005+02:00

This year's Nobel prize for Physics was awarded to Kao, Boyle and Smith. According to the press release, Boyle and Smith "invented the first successful imaging technology using a digital sensor, a CCD". We've all used digital cameras that include such sensors I'm sure, so indeed this was a major achieved. Interestingly, there's a bit of controversy to this award, as there often seems to be.

Touchy 3D UI pixels

2009-10-02T13:17:00.003+02:00

Apple's commercial success of their touch interface has spawned a bunch of companies doing their best to top the Apple's Touch experience. Here's a notable one, which can actually be installed on a Windows machine, given you have a correctly enabled touch panel.

Will the mouse really go away?

One mm thick pixels

2009-09-25T21:26:00.005+02:00

Microsoft's new media player the Zune HD has many new cool features: an all you can eat music license, a touch UI, 720p video and an HD radio. Coolest thing though is the OLED display. This teardown shows it's a mere 1mm thick. That's about what your hair grows in two and a half days.

Unlit pixels: Sony's new OLED walkman

2009-04-17T13:34:00.003+02:00

Sony released a new Walkman with an OLED touch screen. OLEDs don't have a back light, so this player should last a lot longer than players with back-lit LCDs. LCDs typically consume half of the energy of a portable device.

Panoramic pixels

2009-01-16T22:22:00.004+01:00

Out of the blue, my friend Jonah sent me this picture taken from a kite.

1. It's a beautiful picture, taken on a beautiful island, on a beautiful day, at high tide.
2. I spent two summers in the home "De Wokkel" in this picture. Now a few years later a friend of mine who lives a continent away sends me a great aerial photo of the place. Does chance exist?
3. This very sharp picture is taken from a kite, apparently something very hard to do. Were any special techniques like super resolution used? Will we see image stabilization and super resolution techniques become standard features on our cameras soon?
4. There's an interactive panoramic version of this photo available from the author here. It takes a while to load but it's worth the wait.
5. Bonus points if you spot a car in this picture. Visitors can't bring their cars to the Schiermonnikoog island, and most of the locals don't have one either.

Mirrory pixels

2008-12-16T16:18:00.002+01:00

IMEC just announced a new 11 megapixel CMOS-manufactured micro mirror array. Texas Instruments is quite successful with its DLP micro mirrors frequently used in projectors. IMEC now claims twice the pixel density of competing technologies.

Yet another pixel post-processing acquisition

2008-10-24T13:46:00.005+02:00

Sigma Designs acquires Gennum, ST acquires Genesis, Zoran acquires Let It Wave and now IDT acquires Silicon Optix. Silicon Optix was another company that focused on developing the highest quality video format converters. This acquisition is a bit different from the others since IDT doesn't have any digital media application processors for set top boxes or Blu-ray players that need integrated post-processing functionality. Instead, IDT is likely to get synergy out of this acquisition by combining the post-processing products with their mixed signal consumer video products.

Parallel painty pixels

2008-10-20T09:38:00.015+02:00

Every engineer loves Adam and Jamie from the Mythbusters, and now we video engineering guys have a reason to love them even more. Here's a clip showing a display Adam and Jamie built that uses 1100 paintball guns to draw a reproduction of Leonardo's Mona Lisa with a refresh rate of 80ms. They only refresh once, though.

Audio-focused pixels

2008-10-07T12:49:00.008+02:00

Liquid lenses have been around for some time. Varioptic applies a voltage to a lens made of water and oil to change the optical properties and achieve focus or zoom. The Rensselaer Polytechnic Institute created a new optical system using a liquid lens and a small speaker. Instead of applying a voltage, they apply sound to move the water droplets to achieve a focusing effect.

Varioptic hasn't really made an impact on the camera module market yet. Will Rensselaer's technique finally bring liquid lenses into mass production?

Another pixel post-processing company acquired

2008-09-03T14:58:00.007+02:00

Some time ago I wrote about post-processing companies being acquired by larger multimedia companies such that they can address the whole video pipeline, from image capture to display. It's a bit late, but for completeness sake I should really mention that another post-processing company was bought by a bigger multimedia company: Zoran buys Letitwave for $27M. Let it wave had Prof. Stephane Mallat on its staff, a world renowned expert on using wavelets for signal processing.

Watery pixels

2008-08-09T08:59:00.005+02:00

In Canal City, a big mall in Japan, they built a cool new display that drops water in such a pattern that it displays images.

We've now seen wooden pixels, ping pong pixels and watery pixels here. What's next?

Big and expensive pixels

2008-08-06T23:56:00.004+02:00

Today I was at the popular Mediamarkt electronics store in Eindhoven, the Netherlands, where I saw the "largest TV in the world"; a Panasonic 103 inch plasma display. That's well over 2.5 meters diagonal for us metric-centric people. It's got 1920x1080 spatial resolution with 12-bit per color channel resolution. The price tag was also quite hefty at 79,999 Euros ($120,000 dollars). For such an amount you can buy 200 22" flat screen TVs instead. If my math serves me right, that makes for a 580 inch (14.4 meter) diagonal TV. Quite a bit bigger and with a much higher spatial resolution.

I was surprised that the display didn't look more grainy though, even when standing relatively close by. In displays the Megapixel race doesn't follow the same pace as in the image sensor world. Still, with such large screen sizes, will we soon need to capture, store and transmit video with bigger than 1920x1080 resolutions?

Direct pixel manipulation

2008-05-26T16:02:00.005+02:00

The below video shows a new intuitive and simple way of browsing through video material. Instead of browsing through time, by dragging the scrollbar or time bar, you can simply drag objects in the video. The video says it all. Funny side note: first author of the publication is Pierre Dragicevic. More videos and information here. You can even download their free "DimP" player.

Scalable pixel product

2008-05-18T22:27:00.002+02:00

I've spotted the first real SVC product announcement. At NAB, MainConcept, a DivX daughter, presented their SVC implementation. SVC is the new video coding extension to H.264 that doesn't bring higher coding efficiency, but actually worsens it. I wrote about it earlier. The big benefit though is that you can decode parts of the bit stream in case you only need a smaller resolution picture. MainConcept writes "creating an SVC file only causes an approximate 10% file size increase compared to a regular H.264/AVC file".

10% is a lot in video compression.

Thin pixel machinery

2008-05-14T09:34:00.004+02:00

In a recent post I briefly wrote about Sony's new OLED display. One of the main hindrances of market adoption of this technology is cost. But now DuPont and Dainippon have announced they're working together to develop equipment specifically to manufacture OLED displays. The plan is to make machines that basically print the display, using techniques similar to ink jet printers. The ultimate goal is for the OLED displays to achieve LCD price points. My prediction is that within 10 years OLED will have displaced LCD.

Shiny pixel fab being built

2008-05-06T23:20:00.004+02:00

A little while ago I reported about the new IMOD display technology from Qualcomm, which should yield high quality displays that consume very low power. I saw the displays in action at the Mobile World Congress in Barcelona, but was unimpressed. The displays shown there were small, and perhaps similar to the LCDs that displayed the time on my watch about 30 years ago. Very few graphics and no moving pictures were shown. Still, LCDs have come a long way since the seventies, so perhaps the new IMOD displays will have a bright (no pun intended) future also.

What's the next step? High volume, low cost production facilities. Yesterday Qualcomm and Foxlink announced just that. They will jointly build a new dedicated IMOD fab in -- where else -- Taiwan. The fab is expected to be operational in 2009. There's no mention of how many units the fab can produce.

Can Qualcomm, the CDMA wireless communications company, be successful at entering such a new market? Will IMOD take off?

Thin pixels

2008-04-17T14:21:00.006+02:00

Sony Mobile Display showed a 0.2mm-thick 3.5inch OLED display the other day in Tokyo. The resolution is 320x220, and since it is OLED I expect the picture quality to be quite striking. In contrast to LCD, OLED doesn't need a backlight, which means it looks like color-printed paper and is very readable in sunlight.

Lossy versus lossless pixels

2008-04-06T21:18:00.003+02:00

The other day my dad -- who speaks English okay -- thought "lossless" meant "loose less". That's what prompted me to write this entry. There are two different ways of compressing video (and this holds for audio too). Lossless and lossy.

When compressing a video sequence with a lossy method, then decompressing it again, the decompressed images will be close to the original, but not quite the same. Even when using a low compression factor, where the decompressed images can't be distinguished from the original simply by looking at them, there is a difference, which can be computed. Almost all of the video compression methods we use today use such a lossy method, where the least important image data is thrown away.

In the lossless case there's no data lost. The compressed/decompressed images are exactly the same as the original. There's a big penalty though: lossless compression methods don't compress very well. The resulting files take up much more space or bandwidth. Lossless image or video compression methods are still used though, for instance in the medical field, and when storing fingerprint information.

Very very small pixels

2008-03-26T14:58:00.006+01:00

Researchers at Stanford University are developing a multi-aperture image sensor which groups arrays of 16x16 pixels, then puts a tiny lens on each group. Their 3Mpixel image sensor in this way includes a total of 12,616 lenses, compared to a shabby single lens commonly found in cameras. The benefits are plentiful. The simpler electronic design means the pixels can be 0.7um, much smaller than Kodak's 1.4um pixels that I posted about earlier. Camera modules incorporating this technique can be made even smaller, cheaper, more robust, and, most importantly, grab better pictures. Instead of taking a single snapshot, the camera actually takes 12,616 pictures, which can be combined with digital image processing techniques to capture 3D image data, to accurately control depth of field, focus, etc. With enough image processing power available in the camera, this opens up a whole world of new possibilities.

A high level overview of the work can be found here and their technical ISSCC paper can be found here.

Pixels better than real life?

2008-03-09T20:54:00.004+01:00

According to this survey by Motorola, Americans would rather watch the Superbowl on an HDTV than in person. “The survey results really speak to the popularity of high-definition programming,” said Doug Means from Motorola.

That's a lame study and a lame statement. The results of the survey don't say anything about the quality of HD video and how close it gets to being there. Yes, quite a few people would rather sit in their homes than take a plane and sit on a plastic seat for hours watching the game. Yes, a big screen TV presents a much better picture than an old Philco Predicta. But no, nothing compares to being there. And I can say that without having ever been to a superbowl game.

Pixel compression over time

2008-03-07T09:52:00.010+01:00

Here's an interesting graph from Harmonic that I sometimes use in presentations. I often misplace it, so I figured I'd stick it here on this blog. That way I can always find it. The graph shows that video compressors are not all the same. They can be improved over time. This is an important fact for chip makers, since developing a complex chip these days often takes well over a year, and is then sold in the market for a year or so after. The longer you can keep your chip in the market, the more you will sell! If your chip includes a software programmable video subsystem, you can still take advantage of algorithmic improvements, just like Harmonic did, and deliver better video quality.

Very small pixels

2008-03-06T14:43:00.004+01:00

A few weeks ago, Kodak announced their new 5Mpixel image sensor at the Mobile World Congress. The sensor has a 1.4 micron pixel size (1.4 by 1.4 micron) which means the sensor can fit in a 4x4mm camera module. That is about the size of a regular black ant. I am sure a bigger ant could carry such a camera. The Kodak sensor has some novelties. There is a new color filter pattern, which includes a "white" photocell receptor instead of just measuring the amount of red, green and blue. That will require quite some changes to the image processing algorithms. Another novelty is that the sensor measures darkness instead of light. Apparently that can be more accurately implemented in silicon. Like most new sensor introductions, Kodak promises higher quality images than anyone else.

Micron just announced that it spun its image sensor business out into a new company called Aptina. The business will be run by Micron's Bob Gove, who was previously at VLIW processor company Equator. Micron says they have already sampled an even smaller 1.2 micron pixel, which in the same 4x4mm tiny camera module would yield a 7Mpixel sensor.

Pixel etymology

2008-03-04T17:15:00.006+01:00

Did you know the word pixel is derived from "picture element"? Here's a long video that details a search for the history of the pixel, by Richard Lyon. Lots of well known names in the field of video and graphics are mentioned. To skip over the introduction go to 2:20.

Perfect pixel patent

2008-03-04T17:13:00.005+01:00

As early as 1929, Ray Davis Kell described a form of video compression and was granted a patent for it. He wrote, "It has been customary in the past to transmit successive complete images of the transmitted picture. [...] In accordance with this invention, this difficulty is avoided by transmitting only the difference between successive images of the object." Although it would be many years before this technique would actually be used in practice, it is still a cornerstone of many video compression standards today. It's the reason why video using MPEG can be compressed roughly a factor of 10 better than JPEG-compressed still images.

What technique can provide another magnitude of improvement in video compression?

My prediction is that we need to change focus from optimizing for best peak signal to noise performance to optimizing for psycho-visual perception. I.e. "how good do the compressed images look" instead of looking at minimizing the mathematical difference between the original and compressed imagery.

Pixel resolution

2008-02-05T13:55:00.005+01:00

The other day I ran across this very useful resolution chart at Wikipedia:

While not all resolutions I come across are listed (where are QCIF, 176x144, and CIF, 352x288, for instance) and the PAL resolution seems incorrect (they quote 768x576) this is still quite a nice diagram.

Tested pixels

2008-02-04T21:34:00.001+01:00

I just ordered a new camera for personal use. It'll be my first SLR. Most cameras I've held, either in the office or at home, I have pointed to this chart to test the camera:

You can get the original from Stephen Westin in pdf here. Simply printing it on any decent laser printer does a pretty good job. If you have an A3 printer, even better. It's interesting to see that most of today's camera phones don't even do a low-pass filter before subsampling on the viewfinder, causing bad aliasing. There's still lots of room for improvement!

User-generated pixels

2008-02-03T20:21:00.001+01:00

Ever heard of a show called Fun TV with Kato-chan and Ken-chan? Me neither. It was quite successful in Japan in the mid 1980s though and featured some of the first user-generated content. Later, ABC's America's Funniest Home Videos would follow the same recipe of showing slapstick movies that people captured at home with their camcorders. Fast forward to 2005, the year that YouTube was born based on the same principle, but on the internet. In 2007, less than two years later, YouTube was sold for $1.6 billion dollars to Google. Nowadays, over 9 billion videos are watched online per month in the US alone, and YouTube has about 30% of that market. That's quite a lot of user-generated pixels, and for sure a number that will keep on growing for quite some time to come.

Wooden pixels

2008-01-25T12:13:00.002+01:00

After my posts about sluggy pixels and shiny pixels, I think it's only fair to mention the wooden pixels developed in 1999 too. Frame rates are quite acceptable, but there's no color. They even made a wooden mirror out of them. Here's a video on YouTube.

Display pixels versus capture pixels

2008-01-21T21:07:00.002+01:00

One of the artificial questions I've been pondering a bit is what we will see more of: pixels that capture light (cameras) or pixels that make up displays? For several years my prediction was that soon we'd see more cameras than displays in the world. The reasoning was that displays are relatively large and made to be seen by a human. Cameras are tiny though and have many uses. Cameras don't need humans to look at the images captured, they can simply be stored, or analyzed by an algorithm running on a piece of silicon. Since there's only a little over 6 billion of us to view the screens, soon we'd have more cameras than displays.

Some cell-phones include two cameras, one for video conferencing and one for taking snapshots. These only have one display, which confirmed that I was right. Soon we'll have more cameras than displays.

Still, the other day I saw a very small digital photo frame which cost only 15 euro and was meant to be worn on a key chain. It could hold 30 photos or so and contained a tiny battery. This caused me to think that the cross over point of having more cameras world-wide than displays is quite far away. We'll soon have displays on our credit cards, on the outside of our laptops and perhaps even on our clothes.

Do you think we will ever have more cameras than displays?

Pixel compression trends

2008-01-16T17:31:00.002+01:00

I recently ran across this Google Trends utility that keeps track of the number of times that certain terms are used on the web. Click on the image to see the trends of MPEG-2, MPEG-4 and H.264, showing that H.264 clearly passed MPEG-4 in 2007.

Any trends in the world of pixels that you'd like to add? Please leave a link to them into the comment section.

Post-processing pixel companies going away?

2008-01-15T21:47:00.001+01:00

Recently, two chip makers that focused on post-processing were acquired by bigger companies. Sigma Designs acquires Gennum's image processing business, and ST acquires Genesis Microchip. The acquiring companies provide single-chip video processing solutions which include such post-processing functionality, but until now their post-processing wasn't as good as what the smaller focused companies could deliver. The market demands ever increasing picture quality at ever reducing price points, which these acquiring companies are looking to achieve with their acquisitions.

One year ago: analog pixels switched off

2008-01-14T21:28:00.001+01:00

It's been a year since they turned off all over-the-air analog broadcast of TV signals in the Netherlands. I haven't heard a complaint since. Only about 74,000 households picked up the analog signals before, so that was to be expected. The extra bandwidth that became available unfortunately are now used to transmit encoded signals, which you have to pay KPN a monthly fee for to view. In return for the free over the air bandwidth, KPN built and maintains the digital broadcasting masts and systems. Sounds like a pretty good trade for the KPN to me, and a lousy trade for the government and us tax payers. My guess is that it is this monthly fee that is severely reducing the market introduction of digital portable TV receivers for in the car, mobile phones, etc. Even in the home it'd be handy -- when was the last time you pulled a cable through your house?

Do any of you understand why we still have to pay for TV channels transmitted over the air while they contain more than 10 minutes of paid advertising per hour?

Discontinued pixels

2008-01-11T13:56:00.001+01:00

In March of 1993 Jim Blinn, perhaps the ultimate pixel guru, wrote an article called "NTSC: Nice Technology, Super Color". It's a play on what people often say NTSC means: Never The Same Color. The last few sentences of the article read this: "Current plans call for the FCC to adopt a new high-definition television standard some time this year. The FCC will then strongly encourage all broadcasters to switch over to the new standard as soon as possible. By the year 2007, the FCC wants this conversion to HDTV to be complete. Broadcasters will no longer be allowed to use NTSC. Boo hoo.". Well, HD is happening, but NTSC is still alive and will be for quite some time to come.

What's your guess: when will Standard Definition truly die?

Open source pixels

2007-12-12T16:54:00.001+01:00

The other day I noticed a menu option on the iPod Touch called "Legal". Normally, I avoid reading such legal notes, but this time I was curious to see what was mentioned here. Well, a lot was mentioned. It took me 76 "scroll-the-page-down" strokes to reach the bottom of the long list of legal speak. Most interesting fact; more than 3/4 of the notes were open source related! I saw the GPL come by several times for instance.

Anyone dare to estimate how much total effort it took to develop all the software shipped with the iPod Touch, including the effort put into the open source software? Are your SOCs and video subsystems ready to support open source software?

Acquired pixels

2007-12-04T09:45:00.001+01:00

Recently, DivX acquired MainConcept for approximately $22M. MainConcept is a developer of mostly PC-based video codecs, which is the business that DivX is in also.

Which video company will be acquired next?

Shiny pixels at Qualcomm

2007-11-06T12:15:00.001+01:00

CRT, LCD, TFT, OLED, EPD and DLP are just some of the many acronyms used for the techniques behind displays. There's an article in the November 2007 issue of Scientific American that presents a new acronym: IMOD. The IMOD displays are based on many small interferometric modulators, which bounce back light at different intensities. They don't need a backlight, which means power consumption is much lower, ever so important for portable applications. The viewing experience is also greatly enhanced I am sure. The electronic paper displays that I've seen don't use a back-light either and they're great. They read like paper. The e-ink pixels change intensities too slowly to show video though, while the IMOD technology is very fast. The whole technology reminds me of the, also MEMS-based, DLP from Texas Instruments. Within a few years, that technology quickly became prevalent in projectors, beating out LCD.

With better displays, video coding artefacts will only become more apparent. Is your video subsystem ready to capture and play the highest quality video?

Crummy pixels on my iPod Touch

2007-11-02T22:26:00.001+01:00

I recently bought an iPod Touch. The WiFi integration is neat and worked straight out of the box. I now have a pocketable Internet browser, and it even connects directly to YouTube, using the new H.264 codec instead of YouTube's default and inferior Flash codec that the PC-based website uses.

After playing a few videos something interesting happened. The video codec shows very crude artefacts. See the picture below. I haven't found many other iPod users on the web complaining yet, but it's hard to believe I am the only one. Will Apple be able to fix this with a firmware upgrade? If the rumoured Samsung chip at the heart of this device uses a hard-wired video coding subsystem, they likely won't be able to fix the issue quickly in software. Instead, they will have to respin the chip and people will have to return their devices and get new ones months later. Chip inventory will have to be trashed. A reset fixed the issue for me, but it has shown up again.

Are you an SOC designer that still uses hard-wired video codecs? Can you risk designing an SOC that requires a silicon respin to resolve issues that could have been solved in software if a programmable approach had been chosen?

Scalable pixels

2007-10-31T15:05:00.001+01:00

H.264 was standardized quite a while ago in 2003 and brought a "back to basics" compress-video-only approach compared to the feature-laden MPEG-4. So what are the ISO and ITU video standardization gurus working on these days? They're working on SVC, short for scalable video coding. The goal this time is not to achieve a higher coding gain, but instead to make the bitstream scalable.

In SVC, a same single bitstream can be decoded at different resolutions or frame rates. If you're watching the stream on your 1080p big screen TV at home you decode all the bits, and in case you want to play the same stream on your mobile phone, just decode those pixels that you need for the small screen. There are many applications for scalable video coding, but I've seen the concept many times before and people just don't use it. JPEG2000 is scalable. MPEG-4 had a few scalable profiles, even MPEG-2 had scalable extensions. None of them are widely used today. Why? Because there's overhead involved in making a bitstream scalable. A scalable bitstream is larger than a non-scalable one. Also, most system engineers find it simply more practical to just recompress the bitstream for each specific target device. Pixar and Dreamworks even completely re-render their 3D movies dependent on whether it's for the theater or for a DVD. Compressing the resulting video sequence another time doesn't seem that much of an extra burden.

Will things be different this time? Will SVC become a prevalent standard?

Sluggy pixels

2007-10-28T17:28:00.002+01:00

My brother is an artist. One of his friends made this low-resolution and very-low-frame-rate imaging device. It's called the PingPongPixel and displays one image every 2.5 hours. That's about 0.0001 frame per second. Hardly practical, but still pretty nifty. Each pixel is represented by a ping-pong ball, which come in 6 shades of gray.

Anybody know of a display device with an even lower refresh rate?