People who don't have drawing expertise often have beautiful pictures in their minds as the artists do. Magic Brush is a system aiming at assisting users to easily illustrate the pictures in their mind and thus can share them with others. Magic Brush achieved this by letting the users exploit the existing style, color, brush pattern presented in thousands of master pieces or photos avalible via internet.

Texture Synthesis

To do image transfer, besides time concern, there are at least three factors need trade-off. Tile size, border cut width, and the weight of picking the size by reference image or overlap minimum error. One can imagine the low frequency part of image may use larger tiles, medium border cut width, and small weight of reference image. On the other hand, the regions contain features densely, the high frequency parts, may need small tiles, reasonably small border cut and big weight of reference image. In current system, these are fine tune manually and only use single set of parameters for the whole image. We intend to extend this to have the system automatically determine the parameters as a function of reference image(source texture) size, the area users paint, and the frequency within the painting region in the target image. Other ideas such as switch to image analogy method to achieve more flexible brush filter will be test, too.

(Image courtesy of 2001 Aaron Hertzmann, Charles E. Jacobs, Nuria Oliver, Brian Curless, David H. Salesin http://mrl.nyu.edu/projects/image-analogies/tt.html)

Color transfer

Currently, we are using only normalized RGB color space or normalized pixel intensity to do color matching. We intended to extend this to more reasonable methods.There has been several color transfer research has been surveyed. Hertzmann [5] and Haro[4] use the Y channel of the YIQ color space, which in a sense is pixel intensity. However, using only intensity on chromatic images, even under normalization, problems like the tone difference could happen. Even take all three RGB components in to concern and do linear histogram matching [5] could generate results like the right most image in figure below. Reinhard, etc. [8] propose to do color transfer in lαβ color space, which minimizes correlation between channels for many natural scenes. Chang, etc[1] use CIE L*a*b color space and divide it into categories base on human involved experiment, and then do color transfer within different categories. The result of their algorithm is decent. We will want to implement such algorithm in the magic brush system.

(image courtesy of 2001 Erik Reinhard, Michael Ashikhmin, Bruce Gooch, and Peter Shirley [8])

Segmentation

The region users want to apply the brush on could have fuzzy hair boundary like human hair. Even with scalable stroke size, it could be difficult to paint such area. We intend to implement more smart automatic segmentation mechanism like the magic wand in Photoshop. Also, we will create a method to identify which segmented part the system should apply to. This is intend to be done by tracking the orientation of brush motion controlled by the users. For example, the hair border in the image below. When user paint downward, the system apply the brush to hair part, on the other hand when painting upward, apply it to the skin part.

Interaction Design

To do either image transfer base on the texture synthesis methods, color transfer, or brush filtering, the user need to manually tune the parameters such select region from the reference images, specify what to apply (color, brush, texture), further more when doing texture synthesis, parameter such as tile size and border size. Some of these factors are not intuitive to laymen users. As a result the overall interaction experience needs to be carefully designed. For example, we surveyed the I/O Brush [9] system which contains a physical brush with a hidden camera embedded in it. We intend to extend their work to the virtual world by letting user acquire source from whatever one could digitize (real time camera images, photos, art works found on internet).

The preview window idea they introduced could also be used to improve in our system. The colorization method by scribbling on the segmented regions introduced by Levin[6] could also be a good way to apply color transfer.

 

crop and/or wrap to make textures:

generated images (in matlab, manually adjust parameters):

Stack these together with the source image

           

Use stencil buffer to enable paint (show) certain layer. paint with solid circle stroke

Some results:

Tree example:

original green tree:

paint with sparse stroke

Related Links :

[1] Y. CHANG, S. SAITO, K. UCHIKAWA and M. Nakajima. Example-based color stylization of images, ACM Transaction on applied perception, pages 322-245, July 2005.

[2] N. Diakopoulos, I. Essa, and R. Jain. Content Based Image Synthesis. Proceedings of International Conference on Image and Video Retrieval, July 21-23, 2004.

[3]A. A. Efros and W.T. Freeman. Image quilting for texture synthesis and transfer. Proceeding of SIGGRAPH 2001, pages 341-346, August 2001.

[4] A. Haro, and I. Essa. Learning video processing by example. Proceeding of 16th International Conference on Pattern Recognition. Pages 487-491. August 2002.

[5] A. Hertzmann, C. E. Jacobs, N. Oliver, B. Curless, and D. H. Salesin. Image analogies. Proceeding of SIGGRAPH 2001, pages 327-340, August 2001.

[6] A. Levin, D. Lischinski and Y. Weiss. Colorization using optimization. http://www.cs.huji.ac.il/~yweiss/Colorization/

[7] E. Byong Mok Oh, Max Chen, Julie Dorsey and Frédo Durand. Image-Based Modeling and Photo Editing. Proceeding of SIGGRAPH 2001.

[8] E. Reinhard, M. Ashikhmin, B. Gooch, and P. Shirley. Color transfer between images. In IEEE Computer Graphics and Applications 21, pages 34-41. 2001.

[9] K. Ryokai, S. Marti and H. Ishii. Designing the world as your palette. CHI 2005, April 2005.

[10] T. Welsh, M. Ashikhman, and K. Mueller. Transfer color to greyscale images. Proceeding of SIGGRAPH 2002, pages 277-280, August 2002.

Last update: 2005-11-8;