Extract receipt data with Mindee's API using python

 

In this tutorial, you will learn how to parse a receipt image using python. This can help you automatically extract data from receipts without human interaction and use it in your backend applications for many accounting purposes.

 

If you want to get more details about how to use the mindee python library:

• 1. Python SDK overview
• 2. Extracted receipt fields
• 3. Save and restore Receipt objects
• 4. Receipt data validations
• 5. Benchmark Receipt OCR performances

 

Tutorial

 

The final script can be used within any python middleware application or REST API serving your frontend interfaces.

 

In the end of this tutorial, we’ll show you how to highlight the extracted receipt fields on the image using the open source computer vision library OpenCV.

 

The final result looks like this:

 

 

 

API Prerequisites

 

1. You’ll need a free Mindee account. Sign up and confirm your email to login.
2. A receipt.  Look in your bag/wallet for a recent one, download one here  or do a Google Image search for a receipt and download a few to test with.

 

 

Setup the project

 

Create an empty directory wherever you want on your laptop, we’ll call ours “python_receipt’.

We recommend you create a virtual environment before going to the next step, in order to avoid installing the required packages globally on your machine. To do so, run the following command:

 

(macOS and Linux)

python -m venv venv

(windows)

py -m venv venv

 

Then, you'll need to activate your venv, so that pip will install packages in that folder.

 

(macOS and Linux)

source venv/bin/activate

(Windows)

.\env\Scripts\activate

 

 

We need now to Install the mindee python library from PyPi using pip, a package manager for Python.

 

pip install mindee

 

If the install throws an error, you might need to upgrade your version of pip. To do so, run:

 

pip install --upgrade pip 

 

Don't have pip installed? Try installing it, by running this from the command line:

 

$ curl https://bootstrap.pypa.io/get-pip.py | python

 

That’s it for the setup, let’s call the API.

 

 

Call the Mindee receipt API

 

Log in to the platform and click on the following item to access the Expense receipts API:

 

 

If you don’t have an API token yet for this API, go to the “credentials” section and create a new API Key. 

 

 

Create a “main.py” file in your project directory, and paste the following sample code:

 

from mindee import Client

filepath = "/path/to/my/file"
expense_receipt_token = "my-token-here"

mindee_client = Client(expense_receipt_token=expense_receipt_token)


if __name__ == "__main__":
     
     response = mindee_client.parse_receipt(filepath)

     print(response.receipt)

 

 

Replace the “path/to/my/file” placeholder in the code with the path of the receipt image you want to extract data from.

 

Replace the “my-token-here” with the API token you created previously in the platform.

 

You can get back to your console and run the script

 

python main.py

 

You should see the details of the receipt printed in your console:

 

-----Receipt data-----
Filename: receipt_.jpg
Total amount: 10.2 
Date: 2016-02-26
Category: food
Time: 15:20
Merchant name: CLACHAN
Taxes: 1.7; 20.0%; _
Total taxes: 1.7
----------------------

 

Now let’s take a look at the Receipt object.

 

 

Extracted receipt data

 

The API extracts a list of different fields in the receipt (total amount, taxes, date …). 

 

The Receipt object under the Response.receipt attribute has all the features you need to get the receipt data and coordinates in the image to highlight the results.

 

Here is a piece of code to show you how to get some of them:

 

from mindee import Client

filepath = "/path/to/my/file"
expense_receipt_token = "my-token-here"

mindee_client = Client(expense_receipt_token=expense_receipt_token)


if __name__ == "__main__":
     
     response = mindee_client.parse_receipt(filepath)

     # To get the total amount including taxes value (float), ex: 14.24
     print(response.receipt.total_incl.value)
     
     # To get the total amount excluding taxes value (float), ex: 10.21
     print(response.receipt.total_excl.value)
     
     # To get the total tax amount value (float), ex: 8.42
     print(response.receipt.total_tax.value)
     
     # To get the list of taxes
     print(response.receipt.taxes)

     # Loop on each Tax field
     for tax in response.receipt.taxes:
        # To get the tax amount
        print(tax.value)
     
        # To get the tax code for from a tax object
        print(tax.code)
       
        # To get the tax rate
        print(tax.rate)
     
     # To get the receipt date (string)
     print(response.receipt.date.value)
      
     # To get the receipt merchant name (string)
     print(response.receipt.merchant_name.value)
      
     # To get the receipt time (string)
     print(response.receipt.time.value)

 

Run the script and check out the results in your console!

  

Now we are going to add a piece of code that will highlight the features extracted on the pdf. It can help you or your users to very quickly validate the data extraction.

 

 

Highlight features on the image

 

Let’s try to highlight the features as if someone did it with a pen.

 

First, you’ll need to install the computer vision python library OpenCV if you don’t have it already installed in your env. To do so, run:

 

 

pip install opencv-python

 

Each field extracted in the Response.receipt object has a Field.bbox object. It's a list of vertices in relative coordinates representing the bounding box (% of the image width and height).

 

Check what's inside for the total_incl.bbox for example, doing:

 

print(response.receipt.total_incl.bbox)

 

Now that we know how to access the bounding boxes coordinates, we are going to create our highlighter function in which we can feed any field bounding boxes in different steps, as follows:

 

1. Check if the file is a pdf or an image and load it as an OpenCV image in both scenarios
2. Create a mask image
3. Loop on each feature coordinates and draw the feature rectangle on our mask
4. Overlay the mask and original image with alpha
5. Display image to the user

 

Note: each coordinate returned by the API is relative (in % of the image). You'll see there is a relative to absolute conversion in the code.

 

Here is the code step by step:

 

def highlight_features(img_path, coordinates):
   # step 1: Loads the image as an openCV image
   cv_image = cv2.imread(img_path)

   # step 2: create mask image
   overlay = cv_image.copy()
   h, w = cv_image.shape[:2]

   # step 3: Loop on each feature coordinates and draw the feature rectangle on our mask
   for coord in coordinates:
       if len(coord):
           # Create points absolute coordinates
           pt1 = (int(w*coord[0][0]), int(h*coord[0][1]))
           pt2 = (int(w*coord[2][0]), int(h*coord[2][1]))
           cv2.rectangle(overlay, pt1, pt2, (70, 230, 244), cv2.FILLED)

   # step 4: Overlay the mask and original image with alpha
   final_image = cv2.addWeighted(overlay, 0.5, cv_image, 0.5, 0)

   # step 5: Display image to the user
   cv2.imshow("highlighted_image", cv2.resize(final_image, (600, int(600*h/w))))
   cv2.waitKey(0)

 

Finally, we just have to modify the code for executing the highlighting function and printing the results. Here is the final code:

 

 

from mindee import Client
import cv2


def highlight_features(img_path, coordinates):
   # step 1: Load the file as a numpy image     
   cv_image = cv2.imread(img_path)

   # step 2: create mask image
   overlay = cv_image.copy()
   h, w = cv_image.shape[:2]

   # step 3: Loop on each feature coordinates and draw the feature rectangle on our mask
   for coord in coordinates:
       if len(coord):
           pt1 = (int(w*coord[0][0]), int(h*coord[0][1]))
           pt2 = (int(w*coord[2][0]), int(h*coord[2][1]))
           cv2.rectangle(overlay, pt1, pt2, (70, 230, 244), cv2.FILLED)

   # step 4: Overlay the mask and original image with alpha
   final_image = cv2.addWeighted(overlay, 0.5, cv_image, 0.5, 0)

   # step 5: Display image to the user
   cv2.imshow("highlighted_image", cv2.resize(final_image, (600, int(600*h/w))))
   cv2.waitKey(0)


filepath = "/path/to/my/file"
expense_receipt_token = "my-token-here"

mindee_client = Client(expense_receipt_token=expense_receipt_token)


if __name__ == "__main__":
     
     response = mindee_client.parse_receipt(filepath)

     print(response.receipt)

     highlight_features(
        filepath,
        [
            response.receipt.total_incl.bbox,
            response.receipt.time.bbox,
            response.receipt.merchant_name.bbox,
            response.receipt.date.bbox
        ]
    )

 

And the final result!

 

 

Conclusion

 

In just over 1 second, a receipt was uploaded, the API extracted the receipt data, then the result was parsed and highlighted in the image. Cool right?

 

If you wanted to use this kind of script to display results to your users, I'd advise you to do the highlighting in the front-end application, as sending images back from your middleware is not the best option because of payload sizes. The other solution would be to store the final image using cv2.imwrite(...) but it would make your client download the result. To help you with client-side rendering for your images, you can check out our open-source ReactJS SDK.

 

If you have questions, please reach out to us in the chat widget in the bottom right.