Skip to main content
SpringerLink
Log in

Nonlocal Adaptive Biharmonic Regularizer for Image Restoration

  • Published:
Journal of Mathematical Imaging and Vision Aims and scope Submit manuscript

Abstract

In this paper, we propose a nonlocal adaptive biharmonic regularization term for image restoration, combining the advantages of fourth-order models (preserving slopes) and nonlocal methods (preserving textures). Besides the image deblurring and denoising, we apply the proposed nonlocal adaptive biharmonic regularizer to image inpainting, and a weight matrix normalization method is developed to cover the shortage of information loss of the nonlocal weight matrix and accelerate the inpainting process. The existence and uniqueness of the solution are proved. The mathematical property such as mean invariance is discussed. For the numerical solution, we employ the \(L^2\) gradient descent and finite difference methods to design explicit and semi-implicit schemes. Numerical results for image restoration are shown on synthetic images, real images, and texture images. Comparisons with local fourth-order models, nonlocal second-order models, and other state-of-the-art methods are made, which help to illustrate the advantages of the proposed model.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

Notes

  1. https://github.com/ngcthuong/Image-Denoising-Benchmark.

  2. https://math.sjtu.edu.cn/faculty/xqzhang/NLIP_v1.zip.

  3. https://www.ipol.im/pub/art/2017/189/Inpainting_ipol_code.tar.gz.

References

  1. Ambrosio, L., Tortorelli, V.M.: On the approximation of free discontinuity problems. pp. 105–123 (1992)

  2. Atlas, A., Bendahmane, M., Karami, F., Meskine, D., Oubbih, O.: A nonlinear fractional reaction-diffusion system applied to image denoising and decomposition. Discrete Continuous Dyn. Syst. B 26(9), 4963 (2021)

    Article  MathSciNet  MATH  Google Scholar 

  3. Bertalmio, M., Bertozzi, A.L., Sapiro, G.: Navier-Stokes, fluid dynamics, and image and video inpainting. In: Proceedings of the 2001 IEEE Computer society conference on computer vision and pattern recognition. CVPR 2001, 1, I–I (2001) IEEE

  4. Bertalmio, M., Sapiro, G., Caselles, V., Ballester, C.: Image inpainting. In: Proceedings of the 27th annual conference on Computer graphics and interactive techniques, pp. 417–424 (2000)

  5. Bertozzi, A.L., Greer, J.B.: Low-curvature image simplifiers: global regularity of smooth solutions and laplacian limiting schemes. Commun. Pure Appl. Math. J. Issued Courant Inst. Math. Sci. 57(6), 764–790 (2004)

  6. Buades, A., Coll, B., Morel, J.M.: A non-local algorithm for image denoising. In: 2005 IEEE Computer Society conference on computer vision and pattern recognition (CVPR’05), 2, 60–65 (2005) IEEE

  7. Chambolle, A., Lions, P.L.: Image recovery via total variation minimization and related problems. Numer. Math. 76(2), 167–188 (1997)

    Article  MathSciNet  MATH  Google Scholar 

  8. Chan, R.H., Chen, K.: A multilevel algorithm for simultaneously denoising and deblurring images. SIAM J. Sci. Comput. 32(2), 1043–1063 (2010)

    Article  MathSciNet  MATH  Google Scholar 

  9. Chan, T., Marquina, A., Mulet, P.: High-order total variation-based image restoration. SIAM J. Sci. Comput. 22(2), 503–516 (2000)

    Article  MathSciNet  MATH  Google Scholar 

  10. Chen, D., Sun, S., Zhang, C., Chen, Y., Xue, D.: Fractional-order tv-l 2 model for image denoising. Cent. Eur. J. Phys. 11(10), 1414–1422 (2013)

    Google Scholar 

  11. Dabov, K., Foi, A., Katkovnik, V., Egiazarian, K.: Image denoising by sparse 3-D transform-domain collaborative filtering. IEEE Trans. Image Process 16(8), 2080–2095 (2007)

    Article  MathSciNet  Google Scholar 

  12. Danielyan, A., Katkovnik, V., Egiazarian, K.: BM3D frames and variational image deblurring. IEEE Trans. Image Process 21(4), 1715–1728 (2012). https://doi.org/10.1109/TIP.2011.2176954

    Article  MathSciNet  MATH  Google Scholar 

  13. Ding, D., Ram, S., Rodríguez, J.J.: Image inpainting using nonlocal texture matching and nonlinear filtering. IEEE Trans. Image Process 28(4), 1705–1719 (2018)

    Article  MathSciNet  Google Scholar 

  14. Durand, S., Fadili, J., Nikolova, M.: Multiplicative noise removal using L1 fidelity on frame coefficients. J. Math. Imaging Vision 36(3), 201–226 (2010)

    Article  Google Scholar 

  15. Elharrouss, O., Almaadeed, N., Al-Maadeed, S., Akbari, Y.: Image inpainting: a review. Neural Process. Lett. pp. 1–22 (2019)

  16. Fu, S., Zhang, C., Tai, X.: Image denoising and deblurring: non-convex regularization, inverse diffusion and shock filter. Sci. China Inf. Sci. 54(6), 1184–1198 (2011)

    Article  MathSciNet  Google Scholar 

  17. Gilboa, G., Osher, S.: Nonlocal linear image regularization and supervised segmentation. Multiscale Modeling Simul. 6(2), 595–630 (2007)

    Article  MathSciNet  MATH  Google Scholar 

  18. Gilboa, G., Osher, S.: Nonlocal operators with applications to image processing. Multiscale Modeling Simul. 7(3), 1005–1028 (2008)

    Article  MathSciNet  MATH  Google Scholar 

  19. Hajiaboli, M.R.: A self-governing hybrid model for noise removal. In: Pacific-Rim symposium on image and video technology, pp. 295–305. Springer (2009)

  20. Hajiaboli, M.R.: An anisotropic fourth-order diffusion filter for image noise removal. Int. J. Comput. Vision 92(2), 177–191 (2011)

    Article  MathSciNet  MATH  Google Scholar 

  21. Holla Kayyar, S., Jidesh, P.: Non-local total variation regularization approach for image restoration under a poisson degradation. J. Mod. Opt. 65(19), 2231–2242 (2018)

    Article  Google Scholar 

  22. Hou, G., Pan, Z., Wang, G., Yang, H., Duan, J.: An efficient nonlocal variational method with application to underwater image restoration. Neurocomputing 369, 106–121 (2019). https://doi.org/10.1016/j.neucom.2019.08.041. (www.sciencedirect.com/science/article/pii/S0925231219311798)

    Article  Google Scholar 

  23. Jin, Y., Jiang, X., Jiang, W.: An image denoising approach based on adaptive nonlocal total variation. J. Vis. Commun. Image Represent. 65, 102661 (2019). https://doi.org/10.1016/j.jvcir.2019.102661. (www.sciencedirect.com/science/article/pii/S1047320319302822)

    Article  Google Scholar 

  24. Jung, M., Bresson, X., Chan, T.F., Vese, L.A.: Nonlocal Mumford-Shah regularizers for color image restoration. IEEE Trans. Image Process. 20(6), 1583–1598 (2010)

    Article  MathSciNet  MATH  Google Scholar 

  25. Karami, F., Meskine, D., Sadik, K.: A new nonlocal model for the restoration of textured images. J. Appl. Anal. Comput. 9(6), 2070–2095 (2019)

    MathSciNet  MATH  Google Scholar 

  26. Karami, F., Sadik, K., Ziad, L.: A variable exponent nonlocal p(x)-laplacian equation for image restoration. Comput. Math. Appl. 75(2), 534–546 (2018). https://doi.org/10.1016/j.camwa.2017.09.034. (www.sciencedirect.com/science/article/pii/S0898122117305977)

    Article  MathSciNet  MATH  Google Scholar 

  27. Kim, S., Lim, H.: Fourth-order partial differential equations for effective image denoising. Electron. J. Differ. Equ. 17, 107–121 (2009)

    MathSciNet  MATH  Google Scholar 

  28. Laghrib, A., Ghazdali, A., Hakim, A., Raghay, S.: A multi-frame super-resolution using diffusion registration and a nonlocal variational image restoration. Comput. Math. Appl. 72(9), 2535–2548 (2016). https://doi.org/10.1016/j.camwa.2016.09.013. (www.sciencedirect.com/science/article/pii/S0898122116305259)

    Article  MathSciNet  MATH  Google Scholar 

  29. Li, F., Pi, L., Zeng, T.: Explicit coherence enhancing filter with spatial adaptive elliptical kernel. IEEE Signal Process. Lett. 19(9), 555–558 (2012)

    Article  Google Scholar 

  30. Li, P., Li, S.J., Yao, Z.A., Zhang, Z.J.: Two anisotropic fourth-order partial differential equations for image inpainting. IET Image Proc. 7(3), 260–269 (2013)

    Article  MathSciNet  Google Scholar 

  31. Liu, Q., Gao, X., He, L., Lu, W.: Single image dehazing with depth-aware non-local total variation regularization. IEEE Trans. Image Process 27(10), 5178–5191 (2018). https://doi.org/10.1109/TIP.2018.2849928

    Article  MathSciNet  MATH  Google Scholar 

  32. Liu, Q., Zhang, Z., Guo, Z.: On a fractional reaction-diffusion system applied to image decomposition and restoration. Comput. Math. Appl. 78(5), 1739–1751 (2019). https://doi.org/10.1016/j.camwa.2019.05.030. (www.sciencedirect.com/science/article/pii/S0898122119302937. Advances in Fractional Differential Equations (V): Time-space fractional PDEs)

    Article  MathSciNet  MATH  Google Scholar 

  33. Liu, X., Chen, Y.: NLTV-Gabor-based models for image decomposition and denoising. SIViP 14(2), 305–313 (2020). https://doi.org/10.1007/s11760-019-01558-6.

    Article  Google Scholar 

  34. Lou, Y., Zhang, X., Osher, S., Bertozzi, A.: Image recovery via nonlocal operators. J. Sci. Comput. 42(2), 185–197 (2010)

    Article  MathSciNet  MATH  Google Scholar 

  35. Lv, X.G., Song, Y.Z., Wang, S.X., Le, J.: Image restoration with a high-order total variation minimization method. Appl. Math. Model. 37(16), 8210–8224 (2013). https://doi.org/10.1016/j.apm.2013.03.028. (www.sciencedirect.com/science/article/pii/S0307904X13001832)

    Article  MathSciNet  MATH  Google Scholar 

  36. Lysaker, M., Lundervold, A., Tai, X.C.: Noise removal using fourth-order partial differential equation with applications to medical magnetic resonance images in space and time. IEEE Trans. Image Process. 12(12), 1579–1590 (2003)

    Article  MATH  Google Scholar 

  37. Ma, X., Shen, H., Zhao, X., Zhang, L.: Sar image despeckling by the use of variational methods with adaptive nonlocal functionals. IEEE Trans. Geosci. Remote Sens. 54(6), 3421–3435 (2016). https://doi.org/10.1109/TGRS.2016.2517627

    Article  Google Scholar 

  38. Newson, A., Almansa, A., Gousseau, Y., Pérez, P.: Non-local patch-based image inpainting. Image Process. Line 7, 373–385 (2017)

    Article  MathSciNet  Google Scholar 

  39. Nie, X., Qiao, H., Zhang, B., Huang, X.: A nonlocal tv-based variational method for polsar data speckle reduction. IEEE Trans. Image Process 25(6), 2620–2634 (2016). https://doi.org/10.1109/TIP.2016.2552402

    Article  MathSciNet  MATH  Google Scholar 

  40. Papyan, V., Elad, M.: Multi-scale patch-based image restoration. IEEE Trans. Image Process 25(1), 249–261 (2016). https://doi.org/10.1109/TIP.2015.2499698

    Article  MathSciNet  MATH  Google Scholar 

  41. Perona, P., Malik, J.: Scale-space and edge detection using anisotropic diffusion. IEEE Trans. Pattern Anal. Mach. Intell. 12(7), 629–639 (1990)

    Article  Google Scholar 

  42. Rudin, L.I., Osher, S., Fatemi, E.: Nonlinear total variation based noise removal algorithms. Phys. D 60(1–4), 259–268 (1992)

    Article  MathSciNet  MATH  Google Scholar 

  43. Schönlieb, C.B.: Partial differential equation methods for image inpainting. Cambridge University Press, Cambridge (2015). https://doi.org/10.1007/s11760-019-01558-6

    Book  MATH  Google Scholar 

  44. Shah, J.: A common framework for curve evolution, segmentation and anisotropic diffusion. In: Proceedings CVPR IEEE computer society conference on computer vision and pattern recognition, pp. 136–142. IEEE (1996)

  45. Takeda, H., Farsiu, S., Milanfar, P.: Deblurring using regularized locally adaptive kernel regression. IEEE Trans. Image Process. 17(4), 550–563 (2008)

    Article  MathSciNet  Google Scholar 

  46. Wan, W., Liu, J.: Nonlocal patches based Gaussian mixture model for image inpainting. Appl. Math. Model. 87, 317–331 (2020). https://doi.org/10.1016/j.apm.2020.05.030. (www.sciencedirect.com/science/article/pii/S0307904X20302705)

    Article  MathSciNet  MATH  Google Scholar 

  47. Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13(4), 600–612 (2004)

    Article  Google Scholar 

  48. Wen, Y., Vese, L.A.: Nonlocal adaptive biharmonic regularizer for image restoration. In: International symposium on visual computing, pp. 670–681. Springer (2020)

  49. Wen, Y.W., Ng, M.K., Ching, W.K.: Iterative algorithms based on decoupling of deblurring and denoising for image restoration. SIAM J. Sci. Comput. 30(5), 2655–2674 (2008)

    Article  MathSciNet  MATH  Google Scholar 

  50. Xu, J., Feng, A., Hao, Y., Zhang, X., Han, Y.: Image deblurring and denoising by an improved variational model. AEU-Int. J. Electron. C. 70(9), 1128–1133 (2016). https://doi.org/10.1016/j.aeue.2016.05.008. (www.sciencedirect.com/science/article/pii/S1434841116301704)

    Article  Google Scholar 

  51. Yao, W., Guo, Z., Sun, J., Wu, B., Gao, H.: Multiplicative noise removal for texture images based on adaptive anisotropic fractional diffusion equations. SIAM J. Imag. Sci. 12(2), 839–873 (2019)

    Article  MathSciNet  Google Scholar 

  52. You, Y.L., Kaveh, M.: Fourth-order partial differential equations for noise removal. IEEE Trans. Image Process. 9(10), 1723–1730 (2000)

    Article  MathSciNet  MATH  Google Scholar 

  53. Yu, J., Lin, Z., Yang, J., Shen, X., Lu, X., Huang, T.S.: Generative image inpainting with contextual attention. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5505–5514 (2018)

  54. Zhang, J., Chen, K.: A total fractional-order variation model for image restoration with nonhomogeneous boundary conditions and its numerical solution. SIAM J. Imag. Sci. 8(4), 2487–2518 (2015)

    Article  MathSciNet  MATH  Google Scholar 

Download references

Acknowledgements

This work was partially supported by the NSF grant 2012868, the National Natural Science Foundation of China (12001509, 11871133, U21B2075, 11971131, 61873071, 51476047), the Natural Science Foundation of Zhejiang Province (LQ21A010010), Natural Sciences Foundation of Heilongjiang Province (ZD2022A001), the Fundamental Research Funds for the Central Universities (HIT.NSRIF202202), and China Society of Industrial and Applied Mathematics Young Women Applied Mathematics Support Research Project.

Author information

Authors and Affiliations

  1. Harbin Institute of Technology, Harbin, 150001, China

    Ying Wen, Zhichang Guo & Jiebao Sun

  2. University of California, Los Angeles, CA, 90024, USA

    Luminita A. Vese

  3. Department of Mathematics, China Jiliang University, Hangzhou, 310018, China

    Kehan Shi

Authors
  1. Ying Wen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Luminita A. Vese
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Kehan Shi
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Zhichang Guo
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Jiebao Sun
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Zhichang Guo.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This paper is an extension of the published conference paper [48]. Based on the conference paper [48], we add the theoretical analysis of the proposed nonlocal adaptive biharmonic model and the application of image inpainting, and improve the experimental comparison in this paper. The other parts are the same as the conference paper except for some structural adjustments.

Ying Wen thanks Professor Andrea Bertozzi for hosting her for studying at UCLA from September 2019 to October 2020. The authors would like to thank all anonymous referees for their valuable comments and suggestions, and thank Jianlou Xu for sharing their source code.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wen, Y., Vese, L.A., Shi, K. et al. Nonlocal Adaptive Biharmonic Regularizer for Image Restoration. J Math Imaging Vis 65, 453–471 (2023). https://doi.org/10.1007/s10851-022-01129-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10851-022-01129-4

Keywords

Search

Navigation