The following animation shows an intriguing property of motion perception: there exist two attractors (clock-wise and anti-clockwise) in this dynamic system.

Folktales tell you whether you see it clockwise or anticlockwise will determine whether your left brain or right brain  is dominating. What is more interesting to me is how can we have two stable interpretation of the same phenomenon. It clearly shows the existence of multi-stabilities in cognitive system, which is advocated by Gestalt psychologists a long time ago. But the underlying machinery remains elusive – no neuroscientist can explain how the network of neurons can achieve such multi-stabilities.

One useful clue is the existence of multiple attractors in Hopfield network. For example, if x is an attractor, so is -x. The other important clue is the wagon-wheel effect (shown above), which can be viewed as a simplified multi-stable system. This time, the changing speed of green waves makes it easier to appreciate the phase-transition: i.e., from one direction to the opposite (all of a sudden).  Careful inspection shows that when the speed is slow, no uncertainty in our visual perception; as the speed increases (regardless of the direction), it appears our eyes become more and more difficult to catch up with the apparent motion – the interesting thing is that the extreme cases of two opposite directions are indistinguishable, which causes the phase transition. The only sensible explanation for this confusion is the limited sampling resolution of HVS (the same reason as why we can’t visually see hummingbird’s flapping wings – they are way too fast).

This week ,we have discussed wavelet thresholding – an extremely simple operation but has shown effective in image denoising applications. Given the fact that thresholding is among the simplest nonlinear operators, there is a lot we can say about the role of nonlinearity in image processing.

What is wrong with linear models? Linear combination of Gaussian is still Gaussian; linear MMSE estimation of Gaussian is optimal. We have seen so many nice results if we constrain ourselves to the linear regime – why do we want to get out? The scientific reason is: nature does not work by the linear law. As I mentioned in the class, heavy-tail distribution (also refer to pareto distribution) or 80-20 rule have been widely observed for many complex systems in natural and social science. The fundamental flaw with linear laws is they are too “simple” to be responsible for the complexity in natural or social systems. Starting from chaos (e.g., white Gaussian noise), linear filtering alone would not be enough to create any meaningful order. You might start to think of the IIR filter (AR model) we have used for texture synthesis – it takes random noise as input,  outputs some synthetic patterns and the operation appears to be linear (inverse filtering). Those images contain some order but no more than the linear superposition of sines and cosines (e.g., the linear Wiener filtering result we have shown in the class). Even though the linear superposition of wavelet bases has shown much more order than Fourier bases, I personally think it is still far from the truth (one argument I have is that Hilbert space is too small for accounting for complexity).

Then how do we come up with nonlinear models? In addition to thresholding, we have polynomial functions, trigonometric functions, exponential functions, hyperbolic functions … Which one should we choose? Facing the jungle of nonlinear functions, one can’t help wondering if nature is created by some universal principle, it must be simple and elegant. All these names or equations are created by humans not for revealing the essence of nature but to facilitate the inter-person communication (my cliche: mathematics is a language to support logical reasoning). The fundamental law of nonlinearity must be both simple (to describe mathematically) and complex (to capable of accounting for the complexity in nature). Where is this law? I am looking for it and welcome you to join my search.

In the class, I mentioned Translation Invariance (TI) is a difficult concept in image processing. To understand TI, you need to understand down-sampling first; to understand down-sampling, we need to talk about sampling theorem or Analog-Digital conversion. Nyquist-Shannon sampling theorem states the condition for perfect reconstruction of band-limited signals from their discrete samples. The interpolation formula is based on some sinc function (inverse FT of a band-pass filter). Where does TI come from? DSP textbooks tell you the story after AD conversion (i.e., how aliasing is introduced by down-sampling). But the full story starts from the continuous space – i.e., given a pair of AD and DA converters: f(t)->f(n)->\hat{f}(t), how can we assure it is TI?

Unless f(t) is band-limited, the issue of TI is more subtle than you might think. Consider f(t) to be some speech signal and f(t-s) where T is an arbitrary real number. Common sense tells us human auditory system does not distinguish between f(t) and f(t-s) – so it is kind of invariance to translation. But how does HAS achieve this? Imagine you want to build an engineering system (AD+DA) to do the same thing. If a uniform sampling strategy is used (assuming the sampling period is T),  let us look at the maximum (or minimum whichever you like) of f(n) and \hat{f}(t). No matter what kind of linear filtering is used, we can show max[\hat{f}(t)]<=max[f(n)] (mathematically any linear filter h(n) satisfying \sum h(n)=1 is non-expansive). Well then let us compare the maximum of sampled version of f(t) and f(t-s),  we should be able to make they different by twisting s, right? Then asymptotically as T goes zero, we can conclude the maximum of \hat{f}(t) and \hat{f}(t-s)  can’t be the same. But if it the PR, we will end up with max[f(t)] \neq max[f(t-s)]. Contradiction. The fundamental flaw in the attempt to achieve TI is the linearity. I can’t rigorously proof you can’t achieve TI by uniform sampling and linear interpolation; but nature has shown it goes the other way around (nonuniform sampling and nonlinear interpolation).

Convexity is a concept you don’t see in image processing textbooks. It is a little advanced mathematical tool for engineering students. In the mathematical literature, you can refer to Rockafeller’s “Convex Analysis” (comprehensive and deep) and Boyd&Vandenberghe’s “Convex Optimization (online available at http://www.stanford.edu/~boyd/cvxbook/). In the literature of signal processing, Youla’s 1978 paper was likely the fist to introduce convex projection into image restoration. Combette’s 1993 review article on Proc. of IEEE was very readable for engineering students. Projection-based methods have found several successful applications in image processing such as deblurring, inpainting and post-processing. Many other methods such as wavelet thresholding and total-variation (TV) diffusion can also be interpreted as projection onto some convex set.

However, this blog is more about non-convexity than about convexity. Why do we care about non-convexity? As I mentioned in the class, if you think about the collection of all photographic images of the same size, they do not form a convex set because for a pair of images x and y, ax+(1-a)y does not produce another meaningful image (the deeper explanation comes from the lack of superposition principle in the physical world – i.e., two objects cannot occupy the same location in the space). Geometrically it is useful to think about the collection of images as a manifold which is locally isomorphic to a lower-dimensional Euclidean space. How to exploit such nonconvex manifold constraint has been the grand challenge for IP community for many years. Recent advances (including my own work) have shown deterministic annealing (also called graduated non-convexity) is an effective technique for nonconvex optimization. Please refer to the book titled “Visual Reconstruction” and authored by A. Blake for more details (available online and my VIP library should contain a link).

In this week’s computer assignment, many of you faced the obstacle of It turns out only few students with biometrics background knows how to calculate this Receiver-Operational-Curve thing. You might feel disappointed since I never even mentioned the ROC in the class – “how am I supposed to work this out? it is not covered at all!”

Yeah – I admit my responsibility. However, as you might have tasted the flavor of this class, there are many things I intentionally leave out and expect you to learn on your own. I take this approach because I think it is an important step in research and development – you are not always given all you need to solve a problem. When there is a concept you never heard before, just google it and learn; when there is a tool which you are not familiar with (e.g., MEX, LaTex, Perl …), just grab it and play it until you become a master. Since no class can cover ALL useful concepts and tools, why don’t I leave out some and give you a chance to learn yourself.

Learning new concept and tool is never easy. A good tip is to evoke your curiosity – being a curios George is a wonderful thing in learning. When we are small kids, we have great curiosity about the world and that is why we can learn fast. What happened to our curiosity when we grow up? How come so many adults become used to their routines and never willing to touch new things? I don’t know and no one knows. But apparently, the more curious you are, the more you can learn and a better chance you can succeed regardless of your profession – a simple fact that many people tend to forget.

This year’s Nobel Prize in Physics was unexpectedly awarded to three engineers: one is the “father of fiber optics” Charles Kao (a Shanghaiese who was the Chancellor of CUHK) and the other two are inventors of CCD sensors. In the history of Nobel prize, last time engineers got lucky when two Bell Lab engineers accidentally discovered cosmic microwave background radiation (CMBR) in 1964. The discovery of CMBR might be a matter of luck; the invention of fiber optics and CCD sensors are the fruit of perseverance and ingenuity. Before Kao’s first success of fiber optics, information transmission through glass fiber was thought of a crazy idea. It took him great courage and perseverance to find low-loss glass fibers and support the validity of his theory.

The invention of CCD sensors is also quite a story. The original intention of Boyle and Smith was to design a better memory device (not a sensing device). The outcome, called charge bubble device, would have been a failure for information storage applications (the invention of flash memory has to wait much later – by Fujio Masuoka in 1980). However, the genius of Boyle and Smith was to make connection with Einstein’s photoelectric effect and turn a poor memory device into a fabulous information acquisition one. This story clearly shows the “relativity theory” of engineering inventions – a mediocre idea could turn into a miracle as long as you keep an open and curious mind.

In short, it is good to see pioneering work in the profession of EE gets recognized at the highest level – not just deep theories change our view about the world; great inventions and designs could also have a huge impact on our lives. I think this award is a blessing to all engineering students – engineers can also be well respected as long as their work can make an impact.

Based on my interaction with some of you, mathematics and programming skills are likely to be two deciding factors in your success of doing research in various fields (e.g., image processing, biometrics, communication, networking etc.). Therefore, it is inevitable to feel the frustration over some difficult papers (involving deep mathematics) or challenging projects (involving a huge amount of programming works). This blog contains some advices I have collected in the past ten years and want to share with you.

First, you need to decide where is your talent – are you good with math or computers? Math requires strong logical and analytical skills which not everyone can possess – even myself have always admired some other people’s math skills. How can I compete with them? Years of experience tells me that 1) I don’t need to master all mathematical skills; 2) I don’t need to compete but to collaborate. Since I come to believe that analysis, geometry, algebra, statistics and graph theory are all some kind of mathematical languages, I start to realize I do  need to grasp them all. If you can find one language which you feel the most comfortable with, use it to the full strength. I used to keep baffled how come Ingrid Daubechies (the renowned researcher in wavelet theory) was afraid of anything undeterministic. Well it surely explains that you only need to master one language to understand the world. Statistics, which I took for granted when learning it, took me quite some time to understand its limitations and weakness (another lesson of maintaining a healthy skepticism). If you really give up on mathematics, at least find a collaborate who is good at math – I can swear on the effectiveness of such strategy.

Working with a computer are different. It is a safe and powerful tool to help us understand the world (in this class we focus on the image data taken from the real world). MATLAB is one popular language you can master in a short period of time (even MEX or profiling is relatively easy when compared with .Net or PHP or Perl if you have mastered the C language).  For beginners, the test at level 1 is to make sure you can implement your ideas correctly (i.e., you need to know how to use the tool in a proper way). When something weird occurs, you know how to debug and figure out where the problem is. Debugging is an art you can only master after long period of training. After passing this test, you can focus more on how to make the best use of computer to help you understand. If it takes too long to run a program, work with a smaller-size image; if some method does not work as expected, try a different image and play detective; if implementing some paper is too time consuming, search on the web – these days, many resources are available as long as you bother to dig them out. The only risk of using others’ MATLAB codes is that you have to understand them first and sometimes this process could be more time-consuming and frustrating than writing your own codes.

Hope they are helpful.

I understand some of you might be upset about losing some point in CA#2. Now here is the bright side: if you look at the grading system, an A grade is 90/100 – so there is a comfort zone where you can lose as many as 10 points. In fact, usually I also give out some “free” bonus points for best submissions of midterm/final project. So losing a point or two in one assignment should not penalize your overall grade much. Even if you are a perfectionist (wanting a 4.0 GPA), there is really no need to fight for 95 vs. 93 out of 100 since they bear no difference after the final quantization.

What is more important than a few points or an A-level grade is your own progress. Any judgment given by others is less important than your own self-satisfaction. Have I tried hard enough? What have I learned from finishing this assignment? Did I have a better understanding? No matter what is your career objective (research or teaching or development or entrepreneur),  understanding is the most important: A. Einstein becomes the house-held hero because he shows a different perspective of understanding the physical world; R. Feynman is among the greatest teachers (in addition to a brilliant scientist) because he grasps a better understanding about how to communicate with his students (which is one of the black magic I have not understood well); Bill Gates or Steve Jobs become billionaire because they understand the business better than most of other fellows. So you see, the kick is: you only need to understand it (an assignment, a project, a business opportunity) better than your competitors.

How to understand it better? As I said in the class (I was citing R. Hamming’s words), “knowledge scope and productivity is like compound bonds.” The more you know, the more competing edge you have. Nobody knows how we learn, but apparently harder-working people can know more and achieve more than others. I believe every student can understand a known concept or tool given in the textbook if you work hard enough. What is much more challenging is the creation of new concept or tool. As we move into the second month of this class, I will encourage more in-class discussion about the limitations of existing tools. Maintaining a healthy skepticism towards the current-state-of-the-art is a necessary condition for creativity.

Due to time constraint, we will have to say farewell to the topic of autoregressive(AR) model and move to a new chapter today. However, there are still many interesting stuff related to AR you can dig into:

1) Usefulness of AR. As I mentioned in the class, AR model is more successful on speech than image data. Why? First, there is an intuitive physical interpretation for AR modeling of speech signals – a_i’s we obtained through LS method (they are called linear prediction coefficients or LPC) can be viewed as approximation of vocal tract in speech production system. Second, there exists fast algorithm (Levinson-Durbin iteration) to solve LS problem more efficiently by exploiting the Toeplitz property of covariance matrix. More details about LPC and their application in speech coding are referred to my EE467 course. For images, there does not exist physical interpretation yet; part of my PhD thesis is devoted to the study of relationship between AR model and geometric constraint of edges. It is important to note that even spatially-adaptive extension of AR fails for non-edge structures (e.g. corner).

2)There are various extensions of AR models such as ARMA and ARIMA. A good place to dig further is textbooks on time series analysis. Those models have lots of successful applications in finance, which means you can even go to Wall Street if you are really good at those modeling techniques (some EE PhDs indeed take this career path for its apparent lucrative benefit).

3) There are various extension of LS methods including weighted LS (weighting coefficients can be viewed as an adaptive strategy), nonlinear LS (linearity admits closed-form solution; otherwise we have to use numerical method such as Gauss-Newton iterations), total LS (when observation data are noisy). Unfortunately those techniques scatter around in the literature of statistics, linear algebra and numerical analysis. So if you want to try some new idea, it is important that you can find the related papers (didn’t I mention searching is a critical skill in doing re-search).

4) LS is known to be sensitive to outliers. There is growing interest into the magic of l_1 optimization in recent years. To know more, you can use the following keywords in your search: total-variation, basis pursuit, l_1 regularization, compressed sensing.

I was kept busy during the office hours today and glad to see many of you have put significant effort into the first computer assignment. Well, like I said in one of my emails, this assignment might appear easy from a coding (programming) perspective; but there are a lot more that you can learn than just finishing it. Here are some questions I answered and would like to share with the rest of the class:

1. Why did you use MATLAB function “reshape” in the definition of C and y in AR_coeff.m as well as the matlab demo? The underlying meaning of those two lines does not appear apparent.

Answer: Yes, if you start from the scratch to implement LS estimation, you will find the most tricky part is how to define data vector y and matrix C. As soon as you can get them right, the rest is straightforward (and there are several paths you can take). Here is an ad-hoc way of defining y and C: I can simply go through the whole image pixel by pixel; store every pixel into y one-by-one and its corresponding neighbors into matrix C. Such idea is correct except that it runs slow – because you need to loop over every pixel. The codes I gave out is slightly more efficient because it only loops through all the positions in the chosen neighborhood. Of course, to achieve such efficiency, you need to know to convert a 2D matrix into a 1D vector, which is precisely what “reshape” can do. Depending on how you define C and y, the final result a might be a row or column vector. But that doesn’t matter just like your matrix C might look the same as other people’s matrix C^T – many things having different names/labels are in fact the same.

2. For the texture image, why noncausal model gives even errors of larger variance than causal model?

In experiments, something unexpected often contains more information than something expected. In this case, why do we expect noncausal model should work better? Intuitively, is it because it covers all orientations instead of half-plane or because it covers the 4 nearest neighbors instead of 2 distance-1 plus 2 distance-1.414? As soon as you start thinking, you might discover many new issues you have overlooked before. How about I repeat the experiment for the transpose of the image (will I obtain the same result)? I conjecture that the orientation of texture patterns has some impact – how to test such hypothesis? Can you repeat the experiment for the synthesized Gabor filter image (included in my MATLAB demo) and find out the relationship between errors and orientation? So you can see, one leads to another – a seemingly simple assignment could end up with a whole batch of experiments and hypotheses.

Overall, I hope you have found that understanding image data as simple as regular edge or texture pattern is nontrivial at all. Tomorrow, we will refine our AR model to make it more powerful.