Extract receipt data with Mindee's API using python
In this tutorial, you will learn how to parse a receipt image using python. This can help you automatically extract data from receipts without human interaction and use it in your backend applications for many accounting purposes.
If you want to get more details about how to use the mindee python library:
- 1. Python SDK overview
- 2. Extracted receipt fields
- 3. Save and restore Receipt objects
- 4. Receipt data validations
- 5. Benchmark Receipt OCR performances
Tutorial
The final script can be used within any python middleware application or REST API serving your frontend interfaces.
In the end of this tutorial, we’ll show you how to highlight the extracted receipt fields on the image using the open source computer vision library OpenCV.
The final result looks like this:
API Prerequisites
- You’ll need a free Mindee account. Sign up and confirm your email to login.
- A receipt. Look in your bag/wallet for a recent one, download one here or do a Google Image search for a receipt and download a few to test with.
Setup the project
Create an empty directory wherever you want on your laptop, we’ll call ours “python_receipt’.
We recommend you create a virtual environment before going to the next step, in order to avoid installing the required packages globally on your machine. To do so, run the following command:
(macOS and Linux)
python -m venv venv
(windows)
py -m venv venv
Then, you'll need to activate your venv, so that pip will install packages in that folder.
(macOS and Linux)
source venv/bin/activate
(Windows)
.\env\Scripts\activate
We need now to Install the mindee python library from PyPi using pip, a package manager for Python.
pip install mindee
If the install throws an error, you might need to upgrade your version of pip. To do so, run:
pip install --upgrade pip
Don't have pip installed? Try installing it, by running this from the command line:
$ curl https://bootstrap.pypa.io/get-pip.py | python
That’s it for the setup, let’s call the API.
Call the Mindee receipt API
Log in to the platform and click on the following item to access the Expense receipts API:
If you don’t have an API token yet for this API, go to the “credentials” section and create a new API Key.
Create a “main.py” file in your project directory, and paste the following sample code:
from mindee import Client
filepath = "/path/to/my/file"
expense_receipt_token = "my-token-here"
mindee_client = Client(expense_receipt_token=expense_receipt_token)
if __name__ == "__main__":
response = mindee_client.parse_receipt(filepath)
print(response.receipt)
Replace the “path/to/my/file” placeholder in the code with the path of the receipt image you want to extract data from.
Replace the “my-token-here” with the API token you created previously in the platform.
You can get back to your console and run the script
python main.py
You should see the details of the receipt printed in your console:
-----Receipt data-----
Filename: receipt_.jpg
Total amount: 10.2
Date: 2016-02-26
Category: food
Time: 15:20
Merchant name: CLACHAN
Taxes: 1.7; 20.0%; _
Total taxes: 1.7
----------------------
Now let’s take a look at the Receipt object.
Extracted receipt data
The API extracts a list of different fields in the receipt (total amount, taxes, date …).
The Receipt object under the Response.receipt
attribute has all the features you need to get the receipt data and coordinates in the image to highlight the results.
Here is a piece of code to show you how to get some of them:
from mindee import Client
filepath = "/path/to/my/file"
expense_receipt_token = "my-token-here"
mindee_client = Client(expense_receipt_token=expense_receipt_token)
if __name__ == "__main__":
response = mindee_client.parse_receipt(filepath)
# To get the total amount including taxes value (float), ex: 14.24
print(response.receipt.total_incl.value)
# To get the total amount excluding taxes value (float), ex: 10.21
print(response.receipt.total_excl.value)
# To get the total tax amount value (float), ex: 8.42
print(response.receipt.total_tax.value)
# To get the list of taxes
print(response.receipt.taxes)
# Loop on each Tax field
for tax in response.receipt.taxes:
# To get the tax amount
print(tax.value)
# To get the tax code for from a tax object
print(tax.code)
# To get the tax rate
print(tax.rate)
# To get the receipt date (string)
print(response.receipt.date.value)
# To get the receipt merchant name (string)
print(response.receipt.merchant_name.value)
# To get the receipt time (string)
print(response.receipt.time.value)
Run the script and check out the results in your console!
Now we are going to add a piece of code that will highlight the features extracted on the pdf. It can help you or your users to very quickly validate the data extraction.
Highlight features on the image
Let’s try to highlight the features as if someone did it with a pen.
First, you’ll need to install the computer vision python library OpenCV if you don’t have it already installed in your env. To do so, run:
pip install opencv-python
Each field extracted in the Response.receipt
object has a Field.bbox
object. It's a list of vertices in relative coordinates representing the bounding box (% of the image width and height).
Check what's inside for the total_incl.bbox for example, doing:
print(response.receipt.total_incl.bbox)
Now that we know how to access the bounding boxes coordinates, we are going to create our highlighter function in which we can feed any field bounding boxes in different steps, as follows:
- Check if the file is a pdf or an image and load it as an OpenCV image in both scenarios
- Create a mask image
- Loop on each feature coordinates and draw the feature rectangle on our mask
- Overlay the mask and original image with alpha
- Display image to the user
Note: each coordinate returned by the API is relative (in % of the image). You'll see there is a relative to absolute conversion in the code.
Here is the code step by step:
def highlight_features(img_path, coordinates):
# step 1: Loads the image as an openCV image
cv_image = cv2.imread(img_path)
# step 2: create mask image
overlay = cv_image.copy()
h, w = cv_image.shape[:2]
# step 3: Loop on each feature coordinates and draw the feature rectangle on our mask
for coord in coordinates:
if len(coord):
# Create points absolute coordinates
pt1 = (int(w*coord[0][0]), int(h*coord[0][1]))
pt2 = (int(w*coord[2][0]), int(h*coord[2][1]))
cv2.rectangle(overlay, pt1, pt2, (70, 230, 244), cv2.FILLED)
# step 4: Overlay the mask and original image with alpha
final_image = cv2.addWeighted(overlay, 0.5, cv_image, 0.5, 0)
# step 5: Display image to the user
cv2.imshow("highlighted_image", cv2.resize(final_image, (600, int(600*h/w))))
cv2.waitKey(0)
Finally, we just have to modify the code for executing the highlighting function and printing the results. Here is the final code:
from mindee import Client
import cv2
def highlight_features(img_path, coordinates):
# step 1: Load the file as a numpy image
cv_image = cv2.imread(img_path)
# step 2: create mask image
overlay = cv_image.copy()
h, w = cv_image.shape[:2]
# step 3: Loop on each feature coordinates and draw the feature rectangle on our mask
for coord in coordinates:
if len(coord):
pt1 = (int(w*coord[0][0]), int(h*coord[0][1]))
pt2 = (int(w*coord[2][0]), int(h*coord[2][1]))
cv2.rectangle(overlay, pt1, pt2, (70, 230, 244), cv2.FILLED)
# step 4: Overlay the mask and original image with alpha
final_image = cv2.addWeighted(overlay, 0.5, cv_image, 0.5, 0)
# step 5: Display image to the user
cv2.imshow("highlighted_image", cv2.resize(final_image, (600, int(600*h/w))))
cv2.waitKey(0)
filepath = "/path/to/my/file"
expense_receipt_token = "my-token-here"
mindee_client = Client(expense_receipt_token=expense_receipt_token)
if __name__ == "__main__":
response = mindee_client.parse_receipt(filepath)
print(response.receipt)
highlight_features(
filepath,
[
response.receipt.total_incl.bbox,
response.receipt.time.bbox,
response.receipt.merchant_name.bbox,
response.receipt.date.bbox
]
)
And the final result!
Conclusion
In just over 1 second, a receipt was uploaded, the API extracted the receipt data, then the result was parsed and highlighted in the image. Cool right?
If you wanted to use this kind of script to display results to your users, I'd advise you to do the highlighting in the front-end application, as sending images back from your middleware is not the best option because of payload sizes. The other solution would be to store the final image using cv2.imwrite(...) but it would make your client download the result. To help you with client-side rendering for your images, you can check out our open-source ReactJS SDK.
If you have questions, please reach out to us in the chat widget in the bottom right.