cyrildiagne/ar-cutpaste

An AR+ML prototype that enables reducing parts from your environment and pasting them in a image enhancing application.

Even even though most effective Photoshop is being dealt with currently, it’ll additionally handle assorted outputs in the long run.

Demo & extra infos: Thread

⚠️ That is a research prototype and never a consumer / photoshop client application.

Modules

This prototype runs as 3 self sustaining modules:

• The cell app

  • Take a look at out the /app folder for instructions on easy methods to deploy the app to your cell.

• The local server

  • The interface between the cell app and Photoshop.
  • It finds the location pointed on display cloak by the digicam the usage of screenpoint
  • Take a look at out the /server folder for instructions on configuring the local server

• The object detection / background elimination service

  • For now, the salience detection and background elimination are delegated to an exterior service
  • It’d be loads extra efficient to make use of one thing love DeepLap straight throughout the cell app. Nonetheless that hasn’t been applied on this repo yet.

Utilization

1 – Configure Photoshop

• Trot to “Preferences > Poke-ins”, enable “Some distance away Connection” and build a friendly password that you just are going to be wanting later.

2 – Setup the outdoor salience object detection service

• As mentioned above, for the time being, you might want to deploy the
  BASNet model (Qin & al, CVPR 2019) as an exterior HTTP service the usage of this BASNet-HTTP wrapper (requires a CUDA GPU)

• Potentialities are you’ll presumably well need the deployed service URL to configure the local server

3 – Configure and escape the local server

• Apply the instructions in /server to setup & escape the local server.

4 – Configure and escape the cell app

• Apply the instructions in /app to setup & deploy the cell app.

Thanks and Acknowledgements

• BASNet code for ‘BASNet: Boundary-Mindful Salient Object Detection code‘, Xuebin Qin, Zichen Zhang, Chenyang Huang, Chao Gao, Masood Dehghan and Martin Jagersand
• RunwayML for the Photoshop paste code