<h1>Save and restore</h1>

&nbsp;

The&nbsp;<code>mindee.Client</code>&nbsp;includes methods for saving and restoring api calls. Those methods are helpful if you want to perform tests or if you simply want to reload Invoices objects corresponding to your files without performing any additional request.

&nbsp;

&nbsp;

<h2>Save a call</h2>

&nbsp;

The <code>mindee.Response</code> object returned by the <code>mindee.Client</code> when parsing a file is responsible for wrapping all the data from Mindee API calls. It&nbsp;includes a <code>dump</code> method that can be used as follows:

&nbsp;

<pre>
<code class="language-python">from mindee import Client

mindee_client = Client(invoice_token="your_invoices_api_token_here")

invoice_data = mindee_client.parse_invoice(open("./sample.pdf", 'rb'), input_type="file")

invoice_data.dump("./test_dump.json")</code></pre>

&nbsp;

&nbsp;

<h2>Restore a call</h2>

&nbsp;

To restore a client response object, you can use the <code>mindee.Response</code> as follows:

&nbsp;

<pre>
<code class="language-python">from mindee import Response

invoice_data = Response.load("./test_dump.json")</code></pre>

&nbsp;

&nbsp;

Save and restore Mindee Invoice API response with Mindee python helper library

Save and restore

<h1>Extract data from invoices with Mindee&#39;s API using Python&nbsp;</h1>

&nbsp;

In this tutorial, you will learn how to extract data from invoices (pdfs or images) using python. This can help you automatically process invoices in any accounting software.

&nbsp;

If you want to get more details about how to use the mindee python library:

<ul>
	<li>1.&nbsp;<a href="https://mindee.com/documentation/apis/invoice-parsing/python-invoice/getting-started">Getting started with invoice parsing in python</a></li>
	<li>2.&nbsp;<a href="https://mindee.com/documentation/apis/invoice-parsing/python-invoice/multi-pages-invoice-pdfs">Handle multi pages invoices</a></li>
	<li>3.&nbsp;<a href="https://mindee.com/documentation/apis/invoice-parsing/python-invoice/invoice-fields">Extracted invoice fields</a></li>
	<li>4.&nbsp;<a href="https://mindee.com/documentation/apis/invoice-parsing/python-invoice/save-and-restore">Save and restore Invoice objects</a></li>
	<li>5.&nbsp;<a href="https://mindee.com/documentation/apis/invoice-parsing/python-invoice/data-validation">Invoice data validations</a></li>
	<li>6.&nbsp;<a href="https://mindee.com/documentation/apis/invoice-parsing/python-invoice/performances-benchmark">Benchmark Invoice OCR performances</a></li>
</ul>

&nbsp;

&nbsp;

<h2>Tutorial</h2>

&nbsp;

The final script can be used within any python middleware application or REST API serving your frontend interfaces.

&nbsp;

In the end of this tutorial, you will learn how to transform a pdf into an image file, and how to highlight features on it to make human validation easier.&nbsp;

&nbsp;

The final result looks like this:

&nbsp;

<img alt="extract data invoices python mindee api" src="https://mindee-public-website-dev.s3.amazonaws.com/blog/2020/10/29/ezgifcom-gif-maker.gif" />

&nbsp;

&nbsp;

<h2>API Prerequisites</h2>

&nbsp;

<ol>
	<li>You&rsquo;ll need a free Mindee account. <a href="https://platform.mindee.net/signup">Sign up</a> and confirm your email to login.</li>
	<li>An invoice, look into your mailbox, download one <a href="https://mindee-public-website-dev.s3.amazonaws.com/blog/2020/10/29/1p_invoice_b.pdf" target="_blank">here</a> or <a href="https://www.google.com/search?q=sample+invoice">check out on google image</a></li>
</ol>

&nbsp;

&nbsp;

<h2>Setup the project</h2>

&nbsp;

Create an empty directory wherever you want in your laptop, we&rsquo;ll call ours &ldquo;python_invoice&rsquo;.

&nbsp; 
We recommend you to create a virtual environment before going to the next step, in order to avoid installing the required packages globally on your machine.To do so, run the following command:

&nbsp;

(macOS and Linux)

<pre>
<code class="language-bash">python -m venv venv</code></pre>

 
(windows)

<pre>
<code class="language-bash">py -m venv venv</code></pre>

&nbsp;

Then, you&#39;ll need to activate your venv, so that pip will install packages in that folder.

&nbsp;

(macOS and Linux)

<pre>
<code class="language-bash">source venv/bin/activate</code></pre>

 
(Windows)

<pre>
<code class="language-bash">.\env\Scripts\activate</code></pre>

&nbsp;

&nbsp;

We need now to Install mindee python library&nbsp;from PyPi using pip, a package manager for Python.

&nbsp;

<pre>
<code class="language-bash">pip install mindee</code></pre>

&nbsp;

If the install throws an error, you might need to upgrade your version of pip. To do so, run:

&nbsp;

<pre>
<code class="language-bash">pip install --upgrade pip </code></pre>

&nbsp;

Don&#39;t have pip installed? Try installing it, by running this from the command line:

&nbsp;

<pre>
<code class="language-bash">$ curl https://bootstrap.pypa.io/get-pip.py | python</code></pre>

&nbsp;

That&rsquo;s it for the setup, let&rsquo;s call the API.

&nbsp;

&nbsp;

<h2>Call the Mindee Invoice API</h2>

&nbsp;

&nbsp;

Login onto the [Mindee dashboardt](https://platform.mindee.net/) and enter the&nbsp;Invoice API environment by clicking the following card.

&nbsp;

&nbsp;

<img alt="mindee invoice api card" src="https://lh3.googleusercontent.com/JwH29ZumYioDxoUEJ_9BcyZjIMRHq9b-xGf5FWrNS1ZyJ1HKZ66Ihwpxiyop3MB6tNchMr9Py4VbQk2cKQ80nk4Gy1Q0nOe0zs0avA101kBA1uagx9JmTQ2A0FIpC62MxNhi1rKz" style="height:294px; width:300px" />

&nbsp;

&nbsp;

If you don&rsquo;t have an API token yet for this API, go to the &ldquo;credentials&rdquo; section and create a new API Key.&nbsp;

&nbsp;

&nbsp;

<img alt="new api token" src="https://lh3.googleusercontent.com/CWRZZFfY2uVd5Lyx-wxxR_EjWcw6ZBryFhhpNMwP1DBO5LQEEUBfqos2sPjCE4BQ6AdxelSzaybQEl5EGGXDL-eqwr5qXA3dtgn8_TquDVfVp8xFMsjV_npDbTblZ-wXTgOkwoEy" style="height:146px; width:300px" />

&nbsp;

&nbsp;

&nbsp;

Open the &ldquo;main.py&rdquo; file you created a few minutes ago, and paste your sample code.&nbsp; The path to your invoice will be placed in the variable &quot;filepath&quot; since we call it several times. Your main.py file&nbsp;should look&nbsp;like:

&nbsp;

&nbsp;

<pre>
<code class="language-python">from mindee import Client

filepath = "/path/to/your/file"
invoice_token = "my-token-here"

mindee_client = Client(invoice_token=invoice_token)


if __name__ == "__main__":
 
 response = mindee_client.parse_invoice(filepath)

 print(response.invoice)</code></pre>

&nbsp;

&nbsp;

Replace the &ldquo;path/to/my/file&rdquo; placeholder in the code with the path of the invoice pdf or image you want to parse. Replace the &ldquo;my-token-here&rdquo; with the API token you created previously in the platform.

&nbsp;

&nbsp;

You can get back to your console and run the script

&nbsp;

&nbsp;

<pre>
<code class="language-bash">python main.py</code></pre>

&nbsp;

&nbsp;

You should see the details of the invoice appearing in your console:

&nbsp;

<pre>
<code class="language-bash">-----Invoice data-----
Filename: 1p_invoice_b.pdf 
Invoice number: A-1414 
Total amount including taxes: 2608.2 
Total amount excluding taxes: 2415.0 
Invoice date: 2018-09-25
Supplier name: DESIGNS TURNPIKE CO
Taxes: 193.2 8.0%
Total taxes: 193.2
----------------------</code></pre>

&nbsp;

Now let&rsquo;s take a look at the Invoice object.

&nbsp;

<h2>&nbsp;</h2>

<h2>Extracted invoice data</h2>

&nbsp;

The API returns a list of different fields in the invoice (total amount, taxes, date &hellip;). You can find this list in the &ldquo;documentation&rdquo; section in Mindee platform.

&nbsp;

The Invoice object under the&nbsp;<code>Response.invoice</code>&nbsp;attribute has all the feature you need to get the invoice data and coordinates in the image to highlight the results.

&nbsp;

Here is a piece of code to show you how to get some of them:

&nbsp;

<pre>
<code class="language-python">from mindee import Client

filepath = "/path/to/your/file"
invoice_token = "my-token-here"

mindee_client = Client(invoice_token=invoice_token)


if __name__ == "__main__":
 
 response = mindee_client.parse_invoice(filepath)

 # To get the total amount including taxes value (float), ex: 14.24
 print(response.invoice.total_incl.value)
 
 # To get the total amount excluding taxes value (float), ex: 10.21
 print(response.invoice.total_excl.value)
 
 # To get the total tax amount value (float), ex: 8.42
 print(response.invoice.total_tax.value)
 
 # To get the list of taxes
 print(response.invoice.taxes)

 # Loop on each Tax field
 for tax in taxes:
 # To get the tax amount
 print(tax.value)
 
 # To get the tax code for from a tax object
 print(tax.code)
 
 # To get the tax rate
 print(tax.rate)
 
 # To get the invoice date of issuance (string)
 print(response.invoice.invoice_date.value)
 
 # To get the invoice due date (string)
 print(response.invoice.invoice_date.value)</code></pre>

&nbsp;

Run the script and check out the results in your console!

&nbsp;&nbsp;

Now we are going to add a piece of code that will highlight the features extracted on the pdf. It can help you or your users to very quickly validate the data extraction.

&nbsp;

&nbsp;

<h2>Convert pdf into an image</h2>

&nbsp;

If you&rsquo;re using a pdf, we need first to convert the pdf into an image, as it&rsquo;s more convenient to use for highlighting the features on it. If your invoice is an image, you can skip this part.

&nbsp;

To convert the pdf into a jpeg we are going to use the <a href="https://github.com/Belval/pdf2image">pdf2image</a>&nbsp;python library; First install it in your environment:

&nbsp;

<pre>
<code class="language-bash">pip install pdf2image</code></pre>

&nbsp;

Important note: Dealing with pdfs is not easy and this library might not work directly because it sometimes requires some extra dependencies. Check out the github to make sure you installed it correctly.

&nbsp;

Once installed, we are going to create an image out of our pdf:

&nbsp;

&nbsp;

<pre>
<code class="language-python">from pdf2image import convert_from_path, convert_from_bytes
images = convert_from_path(filepath)</code></pre>

&nbsp;

&nbsp;

The images object is an array of Pillow images. Pillow is an image library for python. As we want to use openCV for highlighting the features on the image, we need to convert the pillow image into an OpenCV image. OpenCV is based on the numpy library, and to convert it we are going to use numpy directly.

&nbsp;

If you don&rsquo;t have numpy installed on your environment:

&nbsp;

&nbsp;

<pre>
<code class="language-bash">pip install numpy</code></pre>

&nbsp;

&nbsp;

Now let&rsquo;s convert the Pillow image into an opencv format using numpy:

&nbsp;

&nbsp;

<pre>
<code class="language-ruby">import numpy as np
from pdf2image import convert_from_path
 
def pdf_to_cv_image(filepath, page=0):
 image = convert_from_path(filepath)[page]
 return np.asarray(image)</code></pre>

&nbsp;

 
 
&nbsp;

<h2>Highlight features on the image</h2>

&nbsp;

Let&rsquo;s try to highlight the features as if someone did it with a pen.

&nbsp;

First, you&rsquo;ll need to install the computer vision python library OpenCV if you don&rsquo;t have it already installed in your env. To do so, run:

&nbsp;

&nbsp;

<pre>
<code class="language-bash">pip install opencv-python</code></pre>

&nbsp;

Each field extracted in the <code>Response.invoice</code> object has a <code>Field.bbox</code> object. It&#39;s as simple list of bounding boxes relative vertices (% of the image width and height).

&nbsp;

Check out what&#39;s inside for the total_incl.bbox for example, doing:

&nbsp;

<pre>
<code class="language-python">print(invoice_data.invoice.total_incl.bbox)</code></pre>

&nbsp;

Now that we know how to access the bounding boxes coordinates, we are going to create our highlighter function in which we can feed any field bounding boxes&nbsp;in different steps, as follows:

&nbsp;

<ol>
	<li>Check if the file is a pdf or an image and load it as an openCV image in both scenarios</li>
	<li>Create a mask image</li>
	<li>Loop on each feature coordinates and draw the feature rectangle on our mask</li>
	<li>Overlay the mask and original image with alpha</li>
	<li>Display image to the user</li>
</ol>

&nbsp;

Note: each coordinate returned by the API is relative (in % of the image). You&#39;ll see there is a relative to absolute conversion in the code.

&nbsp;

Here is the code step by step:

&nbsp;

<pre>
<code class="language-ruby">def highlight_features(img_path, coordinates):
 # step 1: Check if the file is a pdf or an image and load it as an openCV image in both scenarios
 if img_path.endswith(".pdf"):
 cv_image = pdf_to_cv_image(img_path)
 else:
 cv_image = cv2.imread(img_path)

 # step 2: create mask image
 overlay = cv_image.copy()
 h, w = cv_image.shape[:2]

 # step 3: Loop on each feature coordinates and draw the feature rectangle on our mask
 for coord in coordinates:
 if len(coord):
 # Create points absolute coordinates
 pt1 = (int(w*coord[0][0]), int(h*coord[0][1]))
 pt2 = (int(w*coord[2][0]), int(h*coord[2][1]))
 cv2.rectangle(overlay, pt1, pt2, (70, 230, 244), cv2.FILLED)

 # step 4: Overlay the mask and original image with alpha
 final_image = cv2.addWeighted(overlay, 0.5, cv_image, 0.5, 0)

 # step 5: Display image to the user
 cv2.imshow("highlghted_image", cv2.resize(final_image, (600, int(600*h/w))))
 cv2.waitKey(0)</code></pre>

 
&nbsp;

Finally, we just have to modify a bit the code for executing the highlighting function and printing the results. Here is what the final code:

&nbsp;

&nbsp;

<pre>
<code class="language-python">from mindee import Client
import cv2
import numpy as np
from pdf2image import convert_from_path


def pdf_to_cv_image(filepath, page=0):
 image = convert_from_path(filepath)[page]
 return np.asarray(image)


def highlight_features(img_path, coordinates):
 # step 1: Check if the file is a pdf or an image and load it as an openCV image in both 
 scenarios
 if img_path.endswith(".pdf"):
 cv_image = pdf_to_cv_image(img_path)
 else:
 cv_image = cv2.imread(img_path)

 # step 2: create mask image
 overlay = cv_image.copy()
 h, w = cv_image.shape[:2]

 # step 3: Loop on each feature coordinates and draw the feature rectangle on our mask
 for coord in coordinates:
 if len(coord):
 pt1 = (int(w*coord[0][0]), int(h*coord[0][1]))
 pt2 = (int(w*coord[2][0]), int(h*coord[2][1]))
 cv2.rectangle(overlay, pt1, pt2, (70, 230, 244), cv2.FILLED)

 # step 4: Overlay the mask and original image with alpha
 final_image = cv2.addWeighted(overlay, 0.5, cv_image, 0.5, 0)

 # step 5: Display image to the user
 cv2.imshow("highlghted_image", cv2.resize(final_image, (600, int(600*h/w))))
 cv2.waitKey(0)


filepath = "/path/to/your/file"
invoice_token = "my-token-here"

mindee_client = Client(invoice_token=invoice_token)


if __name__ == "__main__":
 
 response = mindee_client.parse_invoice(filepath)

 print(response.invoice)

 highlight_features(
 filepath,
 [
 response.invoice.total_incl.bbox,
 response.invoice.total_excl.bbox,
 response.invoice.invoice_number.bbox,
 response.invoice.supplier.bbox,
 response.invoice.invoice_date.bbox,
 response.invoice.due_date.bbox
 ]
 )
</code></pre>

 
&nbsp;

And the final result!

&nbsp;

<img alt="" src="https://mindee-public-website-dev.s3.amazonaws.com/blog/2020/10/29/final_invoice_python.jpg" /> 
&nbsp;

<h2>Conclusion</h2>

&nbsp;

In just over a few seconds, an invoice was parsed and the features were highlighted.&nbsp; 
&nbsp;

If you wanted to use this kind of script to display results to your users, i&#39;d advise you to do the highlighting in the front-end application, as sending images back from your middleware is not the best option because of payload sizes. The other solution would be to store the final image using cv2.imwrite(...) but it would make your client download the result.&nbsp;

&nbsp;

If you have questions, please reach out to us in the chat widget in the bottom right.

Tutorial on how to extract data from invoices (pdfs or images) using python. This can help you automatically parse invoices in any web applications.

<h1>Getting started</h1>

<h2>&nbsp;</h2>

&nbsp;

Get started with invoice parsing in python.

&nbsp;

<h2>1. Install the mindee python helper library</h2>

&nbsp;

Install from PyPi using&nbsp;<a href="https://pip.pypa.io/en/latest/" rel="nofollow">pip</a>, a package manager for Python.

&nbsp;

<pre>
<code class="language-bash">pip install mindee</code></pre>

&nbsp;

Don&#39;t have pip installed? Try installing it, by running this from the command line:

&nbsp;

<pre>
<code class="language-bash">$ curl https://bootstrap.pypa.io/get-pip.py | python</code></pre>

&nbsp;

Getting started with the Mindee API couldn&#39;t be easier. Create a&nbsp;<code>Client</code>&nbsp;and you&#39;re ready to go.

&nbsp;

&nbsp;

&nbsp;

<h2>2. Instantiate your Client</h2>

&nbsp;

The&nbsp;<code>mindee.Client</code>&nbsp;needs your API credentials. You can either pass these directly to the constructor (see the code below) or via environment variables.

&nbsp;

Depending on what type of document you want to parse, you need to add specifics auth token for each endpoint.

&nbsp;

<pre>
<code class="language-python">from mindee import Client

mindee_client = Client(
 invoice_token="your_invoices_api_token_here",
 raise_on_error=True
)</code></pre>

&nbsp;

<ul>
	<li>invoice_token:&nbsp;(string) API key for invoices endpoint</li>
	<li>raise_on_error: (boolean, default True) Specify wheter or not raising an Exception when HTTP errors occur</li>
</ul>

&nbsp;

&nbsp;

We suggest storing your credentials as environment variables. Why? You&#39;ll never have to worry about committing your credentials and accidentally posting them somewhere public.

&nbsp;

&nbsp;

&nbsp;

<h2>3. Parse an invoice</h2>

&nbsp;

Using the parse_invoice&nbsp;method from your client object, you can pass any image or pdf file (see <a href="https://mindee.com/documentation/apis-overview/input-data">inputs file</a>&nbsp;for pdf pages number limits) to get the invoice data.

&nbsp;

<pre>
<code class="language-python">from mindee import Client

mindee_client = Client(
 invoice_token="your_invoices_api_token_here",
 raise_on_error=True
)

parsed_data = mindee_client.parse_invoice("/path/to/file")
</code></pre>

&nbsp;

&nbsp;

<h3>Input types</h3>

&nbsp;

You can pass your input file in three ways:

&nbsp;

From file path

<pre>
<code class="language-python">invoice_data = mindee_client.parse_invoice('/path/to/file', input_type="path")</code></pre>

&nbsp;

&nbsp;From a file object

<pre>
<code class="language-python">with open('/path/to/file', 'rb') as fp:
 invoice_data = mindee_client.parse_invoice(fp, input_type="file")
</code></pre>

&nbsp;

From a base64

<pre>
<code class="language-python">invoice_data = mindee_client.parse_invoice(base64_string, input_type="base64")</code></pre>

&nbsp;

&nbsp;

invoice_data structure

&nbsp;

The invoice_data object returned by the&nbsp;<code>parse_invoice</code>&nbsp;method contains the following elements:

&nbsp;

invoice_data.invoice

&nbsp;

The document attribute&nbsp;is the Invoice object constructed by gathering all the pages into a single document.

&nbsp;

<pre>
<code class="language-python">invoice_data.invoice # returns a unique object from class Invoice</code></pre>

<h4>&nbsp;</h4>

&nbsp;

invoice_data.invoices

&nbsp;

For multi pages pdf, the &#39;pages&#39; attribute is a list of documents objects, each object is constructed using a unique page of the pdf

&nbsp;

<pre>
<code class="language-python">parsed_data.invoices # [Invoice, Invoice ...] </code></pre>

<h4>&nbsp;</h4>

&nbsp;

invoice_data.http_response

&nbsp;

Contains the full Mindee API HTTP response object

&nbsp;

<pre>
<code class="language-python">parsed_data.http_response # full HTTP request object </code></pre>

&nbsp;

&nbsp;

&nbsp;

<h2>4. Display the results</h2>

&nbsp;

You only have to print your invoice object to display the different extracted fields:

&nbsp;

<pre>
<code class="language-python">from mindee import Client

mindee_client = Client(
 invoice_token="your_invoices_api_token_here",
 raise_on_error=True
)

invoice_data = mindee_client.parse_invoice("/path/to/invoice")

print(invoice_data.invoice)</code></pre>

&nbsp;

&nbsp;

Get started with invoice parsing in python and Mindee Invoice API.

Getting started

<h1>Multi pages pdfs</h1>

&nbsp;

You might want to know how to handle multi pages pdfs for many reasons:

&nbsp;

<ol>
	<li>How to handle invoices pdfs with a high number of pages</li>
	<li>How to get a single invoice result for a multi pages pdf invoice</li>
	<li>How to get an invoice result for each page</li>
</ol>

&nbsp;

&nbsp;

<h2>How to handle invoices pdfs with a high number of pages</h2>

&nbsp;

For performances purposes, Mindee APIs don&#39;t allow pdf files as input if they contains too many pages. The <code>parse_invoice</code>&nbsp;method includes a&nbsp; helper for cutting pdf pages in order to perform requests whatever the number of pages in the pdf.

&nbsp;

<pre>
<code class="language-python">from mindee import Client

mindee_client = Client(invoice_token="your_invoices_api_token_here")

invoice_data = mindee_client.parse_invoice("./sample.pdf", cut_pdf=True)</code></pre>

&nbsp;

&nbsp;

cut_pdf: (Bool, default True) If set to True and the input file is a pdf with more than 3 pages, the pdf will be automatically reconstructed as a 3 pages pdf including the first and the two last pags. If set to False, if the pdf input file&nbsp;contains too many pages, an Exception will be thrown.

&nbsp;

&nbsp;

<h2>How to get a single invoice result for a multi pages pdf invoice</h2>

&nbsp;

Each response from mindee parsing methods includes a reconstructed document from all the pages. To construct this object, the <code>mindee.Response</code> class will loop over each page&#39;s result&nbsp;and get for each field the maximum confidence score result. Finally it will construct the Invoice object and you can access it this way:

&nbsp;

<pre>
<code class="language-python">from mindee import Client

mindee_client = Client(invoice_token="your_invoices_api_token_here")

invoice_data = mindee_client.parse_invoice("./3pages.pdf", cut_pdf=True)

single_invoice_for_multipages = invoice_data.invoice</code></pre>

&nbsp;

&nbsp;

&nbsp;

<h2>How to get an invoice result for each page</h2>

&nbsp;

If you know that your multi pages pdf file contains a single invoice for each page, you can still access a&nbsp;<code>mindee.Invoice</code> object for each pages by doing:

&nbsp;

<pre>
<code class="language-python">from mindee import Client

mindee_client = Client(invoice_token="your_invoices_api_token_here")

invoice_data = mindee_client.parse_invoice("./3pages.pdf", cut_pdf=True)

for invoice in invoice_data.invoices:
 print(invoice)</code></pre>

&nbsp;

You might want to know how to handle multi pages invoice pdfs for many reasons, how to handle invoices pdfs with a high number of pages, how to get a single invoice result for a multi pages pdf invoice, how to get an invoice result for each page

Multi pages invoice pdfs

<h1>Invoice&nbsp;fields</h1>

&nbsp;

&nbsp;

<h2>General fields structure</h2>

&nbsp;

Each invoice object contains a set of different fields.

&nbsp;

To acces an invoice object, you need to create a <code>mindee.Client</code>&nbsp;and call the <code>Client.parse_invoice</code>&nbsp;method:

&nbsp;

<pre>
<code>from mindee import Client

# Instantiate your client
mindee_client = Client(invoice_token="your_token_here")

# Call the Mindee invoice endpoint and parse the result
invoice_data = mindee_client.parse_invoice("/path/to/my/file")</code></pre>

&nbsp;

Whatever the field type, each of them contains the four following attributes:

&nbsp;

<ul>
	<li>Value: (Str or Float depending on the field type), corresponds to the field value. Set to None if the field was not extracted.</li>
	<li>Probability: (Float), confidence score of the field prediction.</li>
	<li>bbox:(Array[Float]), contains the relative vertices coordinates of the bounding box containing the field in the image. If the field is not written, the bbox is an empty array.&nbsp;</li>
	<li>reconstructed: (Bool), True if the field was reconstructed using other fields.</li>
</ul>

&nbsp;

Depending on the Field type, there can be extra attributes.

&nbsp;

&nbsp;

&nbsp;

<h2>Total amounts</h2>

&nbsp;

&nbsp;Invoice.total_incl:&nbsp;Total amount including taxes

<pre>
<code class="language-python"># To get the total amount including taxes value (float), ex: 14.24
total_incl = invoice_data.invoice.total_incl.value
</code></pre>

&nbsp;

&nbsp;Invoice.total_excl:&nbsp;Total amount excluding taxes

<pre>
<code class="language-python"># To get the total amount excluding taxes value (float), ex: 10.21
total_excl = invoice_data.invoice.total_excl.value</code></pre>

&nbsp;

Invoice.total_tax: Total tax value reconstructed from tax lines

<pre>
<code class="language-python"># To get the total tax amount value (float), ex: 8.42
total_tax = invoice_data.invoice.total_tax.value</code></pre>

<img src="https://mindee-public-website-dev.s3.amazonaws.com/blog/2020/10/26/image-20201026151427-1.gif" />

&nbsp;

&nbsp;

<h2>Taxes</h2>

&nbsp;

invoice.taxes:&nbsp;Array of Tax fields

Each of the tax field has two extra attributes:

&nbsp;

<ul>
	<li>code: (String), Optional tax code. (HST, GST... for canadian; City Tax, State tax for US etc..)</li>
	<li>rate: (Float), Optional tax rate.</li>
</ul>

&nbsp;

<pre>
<code class="language-python"># To get the list of taxes
taxes = invoice_data.invoice.taxes

# Loop on each Tax field
for tax in taxes:
 # To get the tax amount
 tax_amount = tax.value

 # To get the tax code for from a tax object
 tax_code = tax.code
 
 # To get the tax rate
 tax_rate = tax.rate</code></pre>

&nbsp;

&nbsp;

&nbsp;

<h2>Dates</h2>

&nbsp;

Each date field comes with an extra attribute:

&nbsp;

<ul>
	<li>date_object: (Datetime), datetime object from python datetime.date library</li>
</ul>

&nbsp;

&nbsp;

&nbsp;invoice.invoice_date: Date of issuance of the invoice

<pre>
<code class="language-python"># To get the invoice date of issuance (string)
invoice_date = invoice_data.invoice.invoice_date.value</code></pre>

&nbsp;

invoice.due_date: Payment due date of the invoice

<pre>
<code class="language-python"># To get the invoice due date (string)
due_date = invoice_data.invoice.invoice_date.value</code></pre>

&nbsp;

&nbsp;

&nbsp;

<h2>Supplier informations</h2>

&nbsp;

&nbsp;Invoice.supplier:&nbsp;Supplier name as written in the invoice (logo or supplier infos

<pre>
<code class="language-python"># To get the supplier name
supplier_name = invoice_data.invoice.supplier.value</code></pre>

&nbsp;

&nbsp;Invoice.payment_details:&nbsp;Payment information of the supplier (IBAN only supported yet

<pre>
<code class="language-python"># To get the total amount excluding taxes value (float), ex: 10.21
total_excl = invoice_data.invoice.total_excl.value</code></pre>

&nbsp;

Invoice.vat_number:&nbsp;&nbsp;VAT registration number of the supplier

<pre>
<code class="language-python"># To get the VAT number of the supplier
vat_number = invoice_data.invoice.vat_number.value</code></pre>

&nbsp;

<img src="https://mindee-public-website-dev.s3.amazonaws.com/blog/2020/10/23/image-20201023141445-1.gif" />invoice.company_number:&nbsp;National company number (Only SIRET or SIREN supported yet)

Contains an extra attribute:

&nbsp;

<ul>
	<li>type: (String), Either SIRET or SIREN</li>
</ul>

&nbsp;

<pre>
<code class="language-python"># To get the company number value
company_number = invoice_data.invoice.company_number.value

# To get the company number type
company_number_type = invoice_data.invoice.company_number.type</code></pre>

&nbsp;

&nbsp;

&nbsp;

<h2>Locale and currency</h2>

&nbsp;

locale: Language ISO code

Contains an extra attribute:

&nbsp;

<ul>
	<li>currency: (String), ISO currency code</li>
</ul>

&nbsp;

<pre>
<code class="language-python"># To get the total language code
language = invoice_data.invoice.locale.value

# To get the invoice currency code
currency = invoice_data.invoice.locale.currency</code></pre>

&nbsp;

How to use extracted invoice fields with Mindee python helper library.

Invoice fields

<h1>Validation checks</h1>

&nbsp;

&nbsp;

Each Invoice object comes with a set of validations checks. it can be accessed under the attribute <code>Invoice.checklist</code>.

&nbsp;

Example:

&nbsp;

<pre>
<code class="language-python">from mindee import Client

mindee_client = Client(
 invoice_token="your_invoices_api_token_here",
 raise_on_error=True
)

parsed_data = mindee_client.parse_invoice("/path/to/file")

print(parsed_data.invoice.checklist)</code></pre>

&nbsp;

&nbsp;

This sample code should print something like:

&nbsp;

<pre>
<code class="language-bash">{'taxes_match_total_incl': True, 'taxes_match_total_excl': False, 'taxes_plus_total_excl_match_total_incl': True}</code></pre>

&nbsp;

&nbsp;

taxes_match_total_incl: Try to reconstruct the total amount including taxes with taxes line details&nbsp;and compare the result against the extracted total_incl field.

&nbsp;

taxes_match_total_excl: Try to reconstruct the total amount excluding taxes with&nbsp;taxes line details&nbsp;and compare the result against the extracted total_excl field.

&nbsp;

taxes_match_total_incl: Try to reconstruct the total amount including taxes as the sum of taxes line details amount and the total exluding taxes and compare the result against the extracted total_incl field.&nbsp;

&nbsp;

&nbsp;

&nbsp;

<h2>Interpretation: <code>True</code> checks</h2>

&nbsp;

Mindee&#39;s algorithms extract all fields independantly, whatever the type of document. For example, they don&#39;t use the&nbsp;extracted taxes&nbsp;to reconstruct the total amounts. This way, when trying to reconstruct some fields from others with post-processing and comparing them to raw extracted fields, it&#39;s very unlikely to get a matching reconstruction although the predictions were wrong.

&nbsp;

&nbsp;

<pre>
<code class="language-python">from mindee import Client

mindee_client = Client(invoice_token="your_invoices_api_token_here")

invoice_data = mindee_client.parse_invoice(open("./sample.pdf", 'rb'), input_type="file")

if invoice_data.invoice.checklist["taxes_match_total_incl"] is True:
 # We are sure that taxes and total_incl fields were extracted correctly
 pass

if invoice_data.invoice.checklist["taxes_match_total_excl"] is True:
 # We are sure that taxes and total_excl fields were extracted correctly
 pass

if invoice_data.invoice.checklist["taxes_plus_total_excl_match_total_incl"] is True:
 # We are sure that taxes, total_incl and total_excl fields were extracted correctly
 pass
</code></pre>

&nbsp;

&nbsp;

The&nbsp;<code>mindee</code>&nbsp;library will enforce probabilities to 1 for each field involved in validated check.

&nbsp;

&nbsp;

<h2>Interpretation: <code>False</code> checks</h2>

&nbsp;

The validation checks were done in order to give the user easy way of ensuring maximum precision, but it doesn&#39;t necessarily&nbsp;mean that a <code>False</code>&nbsp;check implies wrong predictions for the involved fields. Here are a few examples leading to impossible checks:

&nbsp;

<ul>
	<li>
	Taxes line details are not always written in invoices, or sometimes they don&#39;t include the tax rate (for example &quot;City Tax&quot;). In this case, as there is no rate extracted by the API and it makes this check impossible to compute.
	</li>
	<li>
	Extra fees with no corresponding taxes are sometimes in the sub-total of the invoice, making this validation impossible.
	</li>
	<li>
	Post total fees (shipping cost ou delivery fees for instance) are excluding from taxes calculation, making the total including impossible to validate.
	</li>
</ul>

&nbsp;

&nbsp;

Built-in invoice data validation rules with Mindee python helper library.

Invoice data validation

<h1>Performances benchmark</h1>

&nbsp;

In a few steps, you can use the mindee python library to perform benchmarks of invoice extraction perfomances against your test dataset.

&nbsp;

<ol>
	<li>Perform api calls for each file in your test set</li>
	<li>Create <code>mindee.Invoice</code> objects from your data (csv, json or whatever)</li>
	<li>Perform the benchmark and analyze the results</li>
</ol>

&nbsp;

&nbsp;

&nbsp;

<h2>1. Perform api calls for each file in your test set</h2>

&nbsp;

Before running the benchmark, we need to collect all the invoice predictions from Mindee API and store them so we can perform the benchmark later without calling the API again.

&nbsp;

To do so, assuming that all your files are stored into a &quot;./test_dataset&quot; folder and that we want to save all the responses into a &quot;./mindee_responses&quot; folder, here is the code:

&nbsp;

<pre>
<code class="language-python">from mindee import Client
import os

TEST_DATA_PATH = "./test_dataset"
RESPONSES_PATH = "./mindee_responses"

mindee_client = Client(invoice_token="your_invoices_api_token_here")

# Loop over all your test files
for test_filename in os.listdir(TEST_DATA_PATH):
 # Get the current file path
 test_file_path = os.path.join(TEST_DATA_PATH, test_filename)
 
 # To make sure we don't stop the process if an error occurs
 try:
 # Parse the current file
 mindee_response = mindee_client.parse_invoice(test_file_path)
 
 # Store the response inside json file to be restored later
 # In this example we use the test file name in the json filename to 
 # be able to retrieve the corresponding file
 response_filepath = os.path.join(RESPONSES_PATH, test_filename+".json")
 mindee_response.dump(response_filepath)
 except Exception as e:
 # In case of error, print the filename so you can understand later
 # what happened
 print(test_filename, e)
</code></pre>

&nbsp;

&nbsp;

&nbsp;

<h2>2. Create <code>mindee.Invoice</code> objects from your data (csv, json,&nbsp;whatever...)</h2>

&nbsp;

The <code>mindee.Invoice</code> class contains a <code>compare()</code> method that takes as inputs two <code>mindee.Invoice</code> objects. Before running our final script, we need now to create a&nbsp;<code>mindee.Invoice object</code> containing the true labels for each fields.

&nbsp;

We&#39;ll use a csv file in this example and the&nbsp;<a href="https://pandas.pydata.org/">pandas</a>&nbsp;library.

&nbsp;

<h3>ground_truth.csv</h3>

<pre>
<code class="language-bash">filename,total_incl,total_excl,invoice_date,invoice_number,due_date,taxes
1.jpg,149.5,115,2020-12-01,F0012020,2021-01-01,11.5-10|23-20</code></pre>

<h3>&nbsp;</h3>

To construct an Invoice object from this dummy csv example, you can simply do:

&nbsp;

<pre>
<code class="language-python">import pandas as pd
from mindee import Invoice

ground_truth_df = pd.read_csv("./ground_truth.csv")


def invoice_from_csv_row(df_row):
 taxes_list = df_row["taxes"].split("|")
 return Invoice(
 total_incl=df_row["total_incl"],
 due_date=df_row["due_date"],
 total_excl=df_row["total_excl"],
 invoice_date=df_row["invoice_date"],
 taxes=[(t.split("-")[0], t.split("-")[1]) for t in taxes_list],
 invoice_number=df_row["invoice_number"]
 )


for index, df_row in ground_truth_df.iterrows():
 invoice_truth = invoice_from_csv_row(df_row)
 print(invoice_truth)
</code></pre>

&nbsp;

&nbsp;

Running this code should print in your console something like this:

&nbsp;

<pre>
<code class="language-bash">-----Invoice data-----
Filename: None 
Invoice number: F0012020 
Total amount including taxes: 149.5 
Total amount excluding taxes: 115.0 
Invoice date: 2020-12-01
Supplier name: None
Taxes: 11.5 10.0%,23.0 20.0%
Total taxes: 34.5
----------------------</code></pre>

&nbsp;

&nbsp;

&nbsp;

<h2>3.&nbsp;Perform the benchmark and see the results</h2>

&nbsp;

Last step, now we need to wrap it all up.

&nbsp;

The Benchmark class has two methods <code>Benchmark.add</code>&nbsp;and <code>Benchmark.save</code>&nbsp;for adding comparison between two Invoice objects, and saving the final metrics.

&nbsp;

&nbsp;

<pre>
<code class="language-python">import pandas as pd
from mindee import Response, Invoice, Benchmark
import os

TEST_DATA_PATH = "./test_dataset"
RESPONSES_PATH = "./mindee_responses"
BENCHMARK_PATH = "./benchmark"

ground_truth_df = pd.read_csv("./ground_truth.csv")
benchmark = Benchmark(BENCHMARK_PATH)


def invoice_from_csv_row(df_row):...
 )


# Loop over each file in our csv
for index, df_row in ground_truth_df.iterrows():
 try:
 # Get test file path
 test_file_path = os.path.join(TEST_DATA_PATH, df_row["filename"])

 # Create ground truth invoice object
 ground_truth_invoice = invoice_from_csv_row(df_row)
 
 # Load the mindee Response for the current file
 mindee_response = Response.load(os.path.join(RESPONSES_PATH, df_row["filename"] + ".json"))

 # Add the comparison between the two invoices to the benchmark
 benchmark.add(
 Invoice.compare(mindee_response.invoice, ground_truth=ground_truth_invoice),
 df_row["filename"]
 )
 
 except Exception as e:
 print(df_row["filename"], e)

benchmark.save()</code></pre>

&nbsp;

&nbsp;

Inside our benchmark folder, you should see a new directory was created, and a metrics.png file shoud have been created inside with the different metrics:

&nbsp;

<img alt="" src="https://mindee-public-website-dev.s3.amazonaws.com/blog/2020/10/26/metrics.png" />

&nbsp;

The Invoice benchmark runs on the 7 fields as shown above. For each of them, you get an information of:

accuracy: the proportion of correct predictions

precision: The proportion of correct predictions among all the non null predictions

Benchmark Invoice OCR performances using Mindee python helper library

Performances benchmark

python

<h1>How to setup your Mindee account</h1>

&nbsp;

&nbsp;

Want to try Mindee APIs? You have already signed up and want to make your first API calls? You&#39;re in the right place !

&nbsp;

&nbsp;

<h2>Signup / login</h2>

&nbsp;

Go to the&nbsp;<a href="https://platform.mindee.net/signup">Signup page</a>&nbsp;to create your account. No credit card is required. You can start using the APIs right away.

&nbsp;

You already have an account ? Go to the&nbsp;<a href="https://platform.mindee.net/login">Login page</a>. If you forgot your password, click&nbsp;<a href="https://platform.mindee.net/password-forgotten">the link</a> and follow the instructions. Check your spam if you don&#39;t receive the email.

&nbsp;

&nbsp;

<h2>Subscribe to an API</h2>

&nbsp;

Once logged in, you land on our API store.

&nbsp;

<img alt="" src="https://mindee-public-website.s3.amazonaws.com/blog/2020/06/11/expense_receipt_card.jpg" />

&nbsp;

Click on the desired API card to subscribe. All our APIs have a free plan.

&nbsp;

&nbsp;

<h2>You are set to go !</h2>

&nbsp;

At this point, everything is set up to use Mindee APIs.

&nbsp;

You can either :&nbsp;

<ol>
	<li>Go to the <a href="https://mindee.com/documentation/get-started/platform-tour">Live Interface</a>&nbsp;to drag and drop images and instantly see the results</li>
	<li>Go to the <a href="https://mindee.com/documentation/get-started/platform-tour">Documentation section</a> and make your first call through the API</li>
</ol>

Set up your account

<h1>Platform tour</h1>

&nbsp;

&nbsp;

<h2>Dashboard</h2>

&nbsp;

From the dashboard section, monitor your API usage.

&nbsp;

&nbsp;

<img alt="" src="https://mindee-public-website.s3.amazonaws.com/blog/2020/06/10/dashboard_article.jpg" />

&nbsp;

Metrics :

 
Traffic: number of inputs processed (images and pdf pages).

&nbsp;

Errors:&nbsp;number of HTTP errors.

&nbsp;

Latency: average server-side computation time in seconds.

&nbsp;

Plan: your current monthly plan. Your default subscription&nbsp;is a free plan that lasts forever and automatically renews each month. Click on the &quot;change plan&quot; button to adjust your subscription according to your needs.&nbsp;While on,&nbsp;your plan will automatically be renewed each month.

&nbsp;

Quota:&nbsp;remaining number of images or pdf pages you can process under your current monthly subscription. 
&nbsp;

Invoicing: find here all your past invoices.

&nbsp;

&nbsp;

<h2>Documentation</h2>

&nbsp;

Everything you need to use the API in your apps.

&nbsp;

<img alt="" src="https://mindee-public-website-staging.s3.amazonaws.com/blog/2020/05/19/documentation_article.jpg" />

&nbsp;

HTTP request: expected HTTP request format.

&nbsp;

Features: specify the JSON response fields, data types&nbsp;and formats.

&nbsp;

Sample code: ready to use sample codes in different languages.&nbsp;Copy and paste in your environment.

&nbsp;

Usage: details about supported input files and technical limitations.

&nbsp;

JSON response: a model JSON response filled with dummy data to navigate more easily the response structure.

&nbsp;

&nbsp;

<h2>Credentials</h2>

&nbsp;

Manage your API access.

&nbsp;

<img alt="" src="https://mindee-public-website.s3.amazonaws.com/blog/2020/06/10/credentials_article.jpg" />

&nbsp;

&nbsp;

API keys : create, revoke and rename your API keys from there. Note that revokation is permanent and cannot be undone.

&nbsp;

&nbsp;

&nbsp;

<h2>Live interface</h2>

&nbsp;

This is your home for no code testing. Drag and drop your own documents to see the API in action.

&nbsp;

Quickly visualize all the extracted fields along with the raw API JSON response.

&nbsp;

<img alt="" src="https://mindee-public-website.s3.amazonaws.com/blog/2020/06/10/live_interface_article.jpg" />

Platform tour

<h1>Make your first request</h1>

&nbsp;

&nbsp;

Prerequisites:&nbsp;for the following, you&#39;ll need a Mindee account and a live API subscription. You can go learn how to create an account and freely subscribe to an API <a href="https://mindee.com/documentation/get-started/setup-your-account">here</a>.

&nbsp;

&nbsp;

<h2>Create an API key</h2>

&nbsp;

Go to the &ldquo;credentials&rdquo; section of your API. Click on &ldquo;add key&rdquo; to create a new token for this API.

&nbsp;

<img alt="" src="https://mindee-public-website.s3.amazonaws.com/blog/2020/06/10/new_token_article.jpg" />

&nbsp;

&nbsp;

&nbsp;

<h2>Get your sample code</h2>

&nbsp;

Go to the Documentation section and find the sample codes in the top right corner:

&nbsp;

<img alt="" src="https://mindee-public-website.s3.amazonaws.com/blog/2020/06/10/sample_code_article.jpg" />

&nbsp;

&nbsp;

&nbsp;

<h2>Call the API</h2>

&nbsp;

The&nbsp;sample codes will help you&nbsp;perform the POST request to Mindee APIs in different languages, and send your inputs the right way.&nbsp;

&nbsp;

For all the sample codes, replace &quot;mindee_api_url&quot; and &quot;your_token_here&quot; placeholders with your own credentials. Everything is available in your API documentation.

&nbsp;

Python

Make sure you have the <a href="https://requests.readthedocs.io/en/master/">requests</a> package installed.&nbsp;If not, you can install it in your env by running :&nbsp;

&nbsp;

<pre>
<code class="language-bash">pip install requests</code></pre>

&nbsp;

<pre>
<code class="language-python">import requests

url = "mindee_api_url"

with open("/path/to/my/file", "rb") as myfile:
 files = {"file": myfile}
 headers = {"X-Inferuser-Token": "your_token_here"}
 response = requests.post(url, files=files, headers=headers)
 print(response.text)</code></pre>

&nbsp;

&nbsp;

cURL

&nbsp;

<pre>
<code>curl -X POST \
 mindee_api_url \
 -H 'X-Inferuser-Token: your_token_here' \
 -H 'content-type: multipart/form-data; boundary=----WebKitFormBoundary7MA4YWxkTrZu0gW' \
 -F file=@/path/to/your/file.png</code></pre>

&nbsp;

&nbsp;

Javascript

Load the following code in an html file and open it with your web browser. The page displays a button allowing you to upload an image. Once uploaded, go check your console, the response is there.

&nbsp;

<pre>
<code class="language-html">&lt;form onsubmit="mindeeSubmit(event)" &gt;
 &lt;input type="file" id="my-file-input" name="file" /&gt;
 &lt;input type="submit" /&gt;
&lt;/form&gt;

&lt;script type="text/javascript"&gt;
const mindeeSubmit = (evt) =&gt; {
 evt.preventDefault()
 let myFileInput = document.getElementById('my-file-input');
 let myFile = myFileInput.files[0]
 if (!myFile) { return }
 let data = new FormData();
 data.append("file", myFile, myFile.name);

 let xhr = new XMLHttpRequest();

 xhr.addEventListener("readystatechange", function () {
 if (this.readyState === 4) {
 console.log(this.responseText);
 }
 });

 xhr.open("POST", "mindee_api_url");
 xhr.setRequestHeader("X-Inferuser-Token", "your_token_here");
 xhr.send(data);
}
&lt;/script&gt;</code></pre>

&nbsp;

You&nbsp;get something like this ;&nbsp;

&nbsp;

<img alt="" src="https://mindee-public-website.s3.amazonaws.com/blog/2020/06/10/html_article.jpg" />

&nbsp;

Ruby

&nbsp;

<pre>
<code class="language-ruby">require 'uri'
require 'net/http'
require 'net/https'
require 'mime/types'

url = URI("mindee_api_url")
file = "/path/to/your/file.png"

http = Net::HTTP.new(url.host, url.port)
http.use_ssl = true
request = Net::HTTP::Post.new(url)
request["content-type"] = 'multipart/form-data; boundary=----WebKitFormBoundary7MA4YWxkTrZu0gW'
request["X-Inferuser-Token"] = 'your_token_here'
request.body = "------WebKitFormBoundary7MA4YWxkTrZu0gW\r\nContent-Disposition: form-data; name=\"file\"; filename=\"#{File.basename(file)}\"\r\nContent-Type: #{MIME::Types.type_for(file)}\r\n\r\n------WebKitFormBoundary7MA4YWxkTrZu0gW--"

response = http.request(request)
puts response.read_body</code></pre>

&nbsp;

Node.js

Make sure you have the&nbsp;<a href="https://www.npmjs.com/package/request">request</a>&nbsp;package installed before running this code.&nbsp;If not, you can install it in your env by running :&nbsp;

&nbsp;

<pre>
<code class="language-bash">npm install request</code></pre>

&nbsp;

<pre>
<code class="language-javascript">const fs = require("fs");
const request = require("request");
const options = {
	method: "POST",
	url: "mindee_api_url",
	headers: {
		"X-Inferuser-Token": "your_token_here",
	},
	formData : {
		"file" : fs.createReadStream("/path/to/your/file.png")
	}
};
request(options, function (err, res, body) {
	if(err) console.log(err);
	console.log(body);
});</code></pre>

&nbsp;

How to make your first request to Mindee APIs

Make your first request

Get started

<h1>&nbsp;Base URL</h1>

&nbsp;

&nbsp;

Prediction APIs are served through a unique endpoint depending on the product you select. They follow this pattern :

&nbsp;

&nbsp;

<pre>
<code class="language-http">https://api.mindee.net/products/&lt;api_name&gt;/&lt;api_version&gt;/predict</code></pre>

&nbsp;

&nbsp;

You need to specify an &lt;api_name&gt; and a &lt;api_version&gt; that corresponds to the API you want to use. You can find what you need in the specific API documentation, see <a href="https://mindee.com/documentation/get-started/platform-tour" style="text-decoration:none">platform tour documentation</a> for more details.

&nbsp;

&nbsp;

<img alt="" src="https://mindee-public-website.s3.amazonaws.com/blog/2020/06/10/base_url_1_article.jpg" />

&nbsp;

<h2>About versionning</h2>

&nbsp;

We release new versions of&nbsp;our APIs for two&nbsp;reasons :&nbsp;

&nbsp;

<ul>
	<li>minor changes: &nbsp;we often release new minor changes of our APIs with new models (faster and&nbsp; / or better), without any loss of service. These releases are transparent on your side as the response structure is untouched.</li>
</ul>

&nbsp;

<ul>
	<li>major changes: this is a major iteration, with possibly no retrocompatibility of the JSON response. Those versions can include new locales, change in the way predictions are formatted... You can simultaneously use several major versions with the same subscription and tokens. You can switch between versions in the dropdown at the top of the page when inside your subscription API area.</li>
</ul>

Base URL

<h1>Authentication</h1>

&nbsp;

You must include a valid token through an HTTP custom header to authenticate your API calls :

&nbsp;

&nbsp;

<pre>
<code>'X-Inferuser-Token: &lt;my-token-here&gt;'</code></pre>

&nbsp;

&nbsp;

For each API, you can create as many tokens as you want in the credentials page. See <a href="https://mindee.com/documentation/get-started/platform-tour" style="text-decoration:none">platform tour documentation </a>for more details.

&nbsp;

&nbsp;

<h2>Token revokation</h2>

&nbsp;

&nbsp;

You can revoke a token on the credentials section. Note that once a token is revoked, you can&#39;t create a new token with the same name, and using this token to authenticate to your API will lead to a 401 error.

&nbsp;

<img alt="" src="https://mindee-public-website.s3.amazonaws.com/blog/2020/06/10/revoke_authentication_article.jpg" />

&nbsp;

<h2>Technical limitations</h2>

&nbsp;

Note that technical limitations are set up for each account depending on your plan. The default limitations are low, even for the free plans, but if you need to lower those limitations, feel free to ask the support team.

&nbsp;

<img alt="" src="https://mindee-public-website.s3.amazonaws.com/blog/2020/06/10/usage_authentication_article.jpg" />

&nbsp;

These limitations are configured for your entire subscription, regardless of tokens or IP address.

How to authenticate your requests to Mindee APIs

Authentication

&nbsp;&nbsp;

<h1>&nbsp;Managing errors</h1>

&nbsp;

Requests can fail for two main reasons:

&nbsp;

Network issues:&nbsp;the server could not be reached

Application error:&nbsp;the server rejected the request

&nbsp;

In the latter case, the error reported by the API&nbsp;contains:

&nbsp;

<pre>
<code class="language-json">{
 "details": "...",
 "message": "optional"
}</code></pre>

&nbsp;

&nbsp;

<table border="1" cellpadding="1" cellspacing="1">
	<thead>
		<tr>
			<th scope="col">Code</th>
			<th scope="col">Possible reasons</th>
		</tr>
	</thead>
	<tbody>
		<tr>
			<td>401 (Unauthorized)</td>
			<td>
			Bad token:&nbsp;you didn&rsquo;t provide the authentication header or the token is wrong.
			</td>
		</tr>
		<tr>
			<td>400 (Bad request)</td>
			<td>
			Wrong file type: file types allowed:&nbsp; png, jpg, webp, pdf

			&nbsp;

			Missing input file: see&nbsp;<a href="https://mindee.com/documentation/apis-overview/input-data">How to manage input files</a>&nbsp;documentation

			&nbsp;

			Too many pages in pdf:&nbsp;APIs have a maximum limit of pages allowed for pdf files

			&nbsp;

			File is too big: APIs have a maximum file size allowed
			</td>
		</tr>
		<tr>
			<td>404 (Not found)</td>
			<td>
			Wrong version: wrong version in base URL
			</td>
		</tr>
		<tr>
			<td>429&nbsp;(Too many requests)</td>
			<td>
			Plan limit reached: you have reached your maximum number of requests.

			&nbsp;

			Too high&nbsp;frequency: number of requests over time is limited. See <a href="https://mindee.com/documentation/apis-overview/authentication">Authentication documentation</a>
			</td>
		</tr>
	</tbody>
</table>

Managing errors

<h1>Input data</h1>

&nbsp;

&nbsp;

Input files must be sent&nbsp;using&nbsp;multipart/form-data encoding with the whole binary file or stream file object, base64 are not supported yet. To see how you can make it work in your language, you can refer to&nbsp;<a href="https://mindee.com/documentation/get-started/make-your-first-request">Make your first request</a>.

&nbsp;

&nbsp;

<h2>Images</h2>

&nbsp;

We support JPG, PNG and WEBP format. Note that our APIs have a maximum file size limitation, think about downsizing your input images if you manipulate heavy files. Always prefer JPG over PDF for scans and photos.

&nbsp;

Tips for images :&nbsp;

&nbsp;

<ol>
	<li>Reduce too high resolution: downscale your image to have a faster upload time and processing time. Because the text can be very small sometimes, resizing the image too much will make it impossible to read. The rule of thumb is that big images should be resized close to 3 megapixels.</li>
	<li>Keep the aspect ratio: when resizing an image, always keep the original aspect ratio.</li>
	<li>Do not upscale: never upscale a low resolution image on your side! This will decrease the algorithm accuracy. It is best to avoid very low resolution images anyhow.</li>
	<li>Do not preprocess images: it is unnecessary to transform your image in black and white, change brightness / contrast if the image looks fine.</li>
	<li>Ask the chat!: If you have any question related to your image input data, feel free to ask the chat to talk to an expert.</li>
</ol>

&nbsp;

<h2>&nbsp;</h2>

<h2>Working with PDFs</h2>

&nbsp;

Our APIs support multi-page PDF files as well. Because we need to convert the PDF pages to actual images, the processing time may be superior compared to images processing.

&nbsp;

Tips for PDFs :&nbsp;

&nbsp;

<ol>
	<li>To be used only if you must: always prefer JPG/PNG format instead of PDF when possible. This is especially true when handling photos and scans. PDF files should be reserved for digital documents only for maximum performance.</li>
	<li>Working with several pages: &nbsp;your PDF may contain several pages. In this scenario, we will handle each page as a separate document. You will therefore receive as many predictions as there are pages. See <a href="https://mindee.com/documentation/apis-overview/json-response-scheme">JSON response scheme</a> documentation for more details.</li>
	<li>Limitations: here is a maximum number of pages you can send, check the Documentation page of your selected API.</li>
</ol>

Input data best practices for Mindee APIs

Input data

<h1>JSON response scheme</h1>

&nbsp;

Each API response follows the same response scheme. The predictions content is specific with the API you use. Once you have subscribed to your preferred API, read the documentation to check for all the outputs available.

&nbsp;

<pre>
<code class="language-json">{
 "call": {
 "endpoint": {
 "name": "endpoint", 
 "version": "2.1"
 },
 "finished_at": "2017-04-11T02:30:49+00:00", 
 "id": "2a81a403-0389-4b39-9b1f-22b5febf795e", 
 "n_documents": 1, 
 "n_inputs": 1, 
 "processing_time": 0.725, 
 "started_at": "2017-04-11T02:30:49+00:00"
 }, 
 "documents": [
 {
 "id": "bf53e6b1-1d29-4187-97ec-bab10eca5744", 
 "name": "filename.png"
 }
 ],
 "predictions": [

 ...

 ]
}
</code></pre>

&nbsp;

call:&nbsp;metadata about the request

&nbsp;

id: a UUID - this identifies your http call

processing_time: server side processing time in seconds

started_at: datetime for when the call has been received

finished_at: datetime the call has been fully processed

n_documents:&nbsp; number of documents processed.

n_inputs: number of images and/or pdf pages processed. Your quota will be decreased by this value.

endpoint:

&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; name: name of the endpoint used to make the call

&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; version: selected API version

&nbsp;

&nbsp;

documents&nbsp;:&nbsp;array with the details of each document sent to the API. NB: a pdf page counts for one document.

id: a UUID - this identifies your document

name: name of file sent, for a pdf this may add the page number at the end

&nbsp;

predictions: Array with the predictions for each document processed. The structure inside each element of this array is specific with the API you use. NB: for now, each pdf page will output a prediction

General JSON response scheme for Mindee APIs

JSON response scheme

APIs overview

<h1>Parsing receipts with Mindee&#39;s API</h1>

&nbsp;

Anyone who has filed an expense report can tell you: receipt tracking and expense logging is a headache.&nbsp; Enter Mindee&rsquo;s receipt parsing API, which uses deep learning to automatically, accurately and instantaneously parse your receipt details.

&nbsp;

In this tutorial, we will walk through the steps to use Mindee&rsquo;s Receipt Parsing API.&nbsp; Let&rsquo;s get started!&nbsp;

&nbsp;

<h2>API Prerequisites</h2>

<ol>
	<li>You&rsquo;ll need a free Mindee account. <a href="https://platform.mindee.net/signup" target="_blank">Sign up</a>&nbsp;and confirm your email to login.</li>
	<li>A receipt.&nbsp; Look in your bag/wallet for a recent one, or do a <a href="https://www.google.com/search?q=cafe+receipt" target="_blank">Google Image search</a> for a receipt and download a few to test with.</li>
</ol>

&nbsp;

&nbsp;

&nbsp;<img src="https://lh3.googleusercontent.com/Dlw1124fbpDBiZamK8ohelvR9MNK0UlnKutB9UDTfp4vD6FzNN26sXwlkJUFGhiTNvAW_PxWNFz98dGSicLm8qkrEey7VVBY1AE_bErDOKLILhnFGL3UzEmvVetBfad-70qmSXLG" />

&nbsp;

<h2>Setting up the API</h2>

&nbsp;

Log into you Mindee account and access your Expense Receipt API environment by clicking the Expense Receipts card:

&nbsp;

<img src="https://lh6.googleusercontent.com/7V0dnGLFfrvOHViat-kxpTs888k2C3Nk1AJlSlDGks9U5tJD2DjWXXMzC0ks-BcmfRR4-gfKILmhtLesNoMrdfRBjb6OmySZALeqeuUNTBEDnLc_87Kc-vkRYbVKhGjmh4bTKQx9" />

&nbsp;

To activate the API, click the &ldquo;Try for Free&rdquo; button to access our generous free tier. You&rsquo;ll land on the dashboard page - where you can quickly see API usage (you have none right now, but that will change).&nbsp; On the left navigation, there are links to &ldquo;Documentation&rdquo;, &ldquo;Credentials&rdquo; and &ldquo;Live Interface&rdquo;.&nbsp; The docs tab has all of the technical details you&rsquo;ll need to build for the receipts API endpoint, and the Live Interface is a cool interactive demo. Rather than try out the demo, we want to build with the API,&nbsp; so click on &ldquo;Credentials&rdquo; to create an API token.

&nbsp;

Add a new token. In this example, I&rsquo;ve named it &ldquo;Tutorial&rdquo;:

&nbsp;

<img src="https://mindee-public-website.s3.amazonaws.com/blog/2020/09/01/pasted-image-0-1.png" style="height:247px; width:500px" />

&nbsp;

Click &ldquo;Add New Key&rdquo; and you&rsquo;ll be able to see your API token,.

&nbsp;

Now, we are ready to make an API call.&nbsp; In this example, we&rsquo;ll be using cURL.

&nbsp;

<pre>
<code class="language-bash">curl -X POST \ https://api.mindee.net/products/expense_receipts/v2/predict \ 

-H 'X-Inferuser-Token: {apiToken}’ \

-H 'content-type: multipart/form-data; boundary=----WebKitFormBoundary7MA4YWxkTrZu0gW' \

-F file=@/path/to/your/file.png</code></pre>

&nbsp;

Simply replace {apiToken}&nbsp;with your new API token and /path/to/your/file/png with the path to your receipt.&nbsp;

NOTE: You can also copy this code right from the documentation tab of the API with your API token inserted for you.

&nbsp;

In this example, I used a receipt from the grocery store in Koln airport (my last business trip in 2020):

&nbsp;

<img src="https://mindee-public-website.s3.amazonaws.com/blog/2020/09/03/comporessed_doug_receipt.jpg" style="height:717px; width:500px" />

&nbsp;

Pasting the cURL sample into my terminal, I hit enter and about a second later, I received a JSON response with the receipt details.&nbsp; The full JSON can be accessed <a href="https://mindee-public-website.s3.amazonaws.com/blog/2020/09/01/rewe.json">here</a>.&nbsp; Since the response is quite verbose, we will walk through the various fields section by section.

&nbsp;

&nbsp;

<h2>API Response: Parsing Results</h2>

&nbsp;

<h3>Summary &amp; Documents section:</h3>

&nbsp;

The first two sections of the response contain information about the API call made:

&nbsp;

<pre>
<code class="language-json">"call": {
    "endpoint": {
        "name": "expense_receipts",
        "version": "2.1"
    },
    "finished_at": "2020-08-29T18:01:22+00:00",
    "id": "e47d8654-0df7-4839-a282-2c04bf293886",
    "n_documents": 1,
    "n_inputs": 1,
    "processing_time": 1.087,
    "started_at": "2020-08-29T18:01:21+00:00"
},
"documents": [
    {
    "id": "66d9adc6-76cf-4c42-8622-3dadb660ac32",
    "name": "IMG_20200301_073354.jpg"
    }
],</code></pre>

&nbsp;

The call section tells us that we ran on the expense receipts endpoint, uploading one document that is one page long, and after just about 1 second, the file was processed.&nbsp; The documents section gives the Mindee id for the file, and the filename.

&nbsp;

<h2>Predictions:&nbsp;&nbsp;</h2>

&nbsp;

Now we are getting to the exciting stuff.&nbsp; The Predictions section is broken into several sections.&nbsp; Several of these are identifying fields on the receipt, and others are using Machine Learning to deduce information from the receipt.&nbsp; Let&rsquo;s go through each section:

<h3>&nbsp;</h3>

<h3>Category</h3>

&nbsp;

<pre>
<code class="language-json">category": {
     "probability": 0.51,
     "value": "miscellaneous"
},</code></pre>

&nbsp;

The API make a prediction on the type of purchase.&nbsp; In this case, it is 51% sure it is miscellaneous.&nbsp; The possible categories are [toll, food, parking, transport, accommodation, gasoline, miscellaneous].

&nbsp;

<h3>Date</h3>

&nbsp;

Identified from text on&nbsp;the receipt and converted into ISO format. This purchase was made on February 3, 2020, and the model is 99% confident in that choice.&nbsp; The segmentation bounding box provides 4 (x,y) coordinates indicating where the date was pulled from the receipt [(0,0) is the upper left corner, (1,1) is the bottom right corner].

&nbsp;

<pre>
<code class="language-json">"date": {
    "iso": "2020-02-03",
    "probability": 0.99,
    "raw": "03-02-2020",
    "segmentation": {
    "bounding_box": [
      [0.64,0.661],
      [0.801,0.661],
      [0.801,0.686],
      [0.64,0.686]
  ]
}</code></pre>

&nbsp;

<h3>Locale</h3>

&nbsp;

Using data from the&nbsp;receipt, the API can predict where the purchase was made, the language and the currency:. Check the documentation for the latest support. At the time of writing, support is centered on Europe and North America.

&nbsp;

<pre>
<code class="language-json">"locale": {
    "country": "DE",
    "currency": "EUR",
    "language": "de",
    "probability": 0.77,
    "value": "de-DE"
},</code></pre>

&nbsp;

In the case of my receipt, it is 77% confident that the purchase is in German, made in Germany, and in euros.

&nbsp;

<h3>Merchant</h3>

&nbsp;

<pre>
<code class="language-json">"merchant": {
    "name": "REWE",
    "probability": 0.91,
    "segmentation": {
    "bounding_box": [
      [0.279, 0.135],
      [0.719, 0.135],
      [0.719,0.23],
      [0.279,0.23]
     ]
}</code></pre>

The API correctly predicted (it was 91% sure) that it was a REWE store. Again, four (x,y) points mark the location of the text naming the Merchant on the image.

&nbsp;

<h3>Orientation</h3>

&nbsp;

<pre>
<code class="language-json">"orientation": {
   "degrees":0,
   "probability": 0.99
},</code></pre>

Did the document require rotation before parsing?&nbsp; Measured in 90 degree increments [0.90.180.270]. In&nbsp;this case, it did not require any rotation.

&nbsp;

<h3>Taxes</h3>

&nbsp;

<pre>
<code class="language-json">"taxes": [],</code></pre>

&nbsp;

If any taxes are identified in the receipt, the will appear here.&nbsp; In this case, no taxes&nbsp;were found (but this is the correct result).

<h2>&nbsp;</h2>

<h3>Time</h3>

&nbsp;

<pre>
<code class="language-json">"time": {
    "iso": "15:50",
    "probability": 0.99,
    "raw": "15:50",
    "segmentation": {
    "bounding_box": [
      [0.649,0.898],
      [0.732,0.898],
      [0.732,0.925],
      [0.649,0.925]
    ]
}</code></pre>

&nbsp;

Time the receipt was printed, confidence, and the (x,y) coordinates that bound the field in the image.

<h2>&nbsp;</h2>

<h3>Total</h3>

&nbsp;

Perhaps the most important part of the receipt, the total spent, along with confidence and the box indicating the location on the receipt.

&nbsp;

<pre>
<code class="language-json">"total": {
    "amount": 17.74,
    "probability": 0.99,
    "segmentation": {
    "bounding_box": [
     [0.663,0.589],
     [0.765,0.589],
     [0.765,0.617],
     [0.663,0.617]
   ]
}</code></pre>

&nbsp;

<h2>&nbsp;</h2>

<h2>Summary:</h2>

&nbsp;

In just over 1 second, a receipt was uploaded, parsed and the response returned to the end user.&nbsp; We know that &euro;17.74 was spent at a REWE in Germany on February 3, 2020 at 15:50.&nbsp; Using the bounding boxes, we can have the user validate the values, and then input this data into an expense management system.&nbsp;&nbsp;

&nbsp;

&nbsp;

<h2>Conclusion</h2>

&nbsp;

Using the MIndee receipt parsing API, you can quickly validate receipts, allowing for faster, more accurate (and less painful) expense management for our users. If you have questions, please reach out to us in the chat widget in the bottom right.

Anyone who has filed an expense report can tell you: receipt tracking and expense logging is a headache.  Enter Mindee’s receipt parsing API, which uses deep learning to automatically, accurately and instantaneously parse your receipt details. In this tutorial, we will walk through the steps to use Mindee’s Receipt Parsing API.  Let’s get started!

<h1>Extract receipt data with Mindee&#39;s API using python</h1>

&nbsp;

In this tutorial, you will learn how to parse a receipt image using python. This can help you automatically extract&nbsp;data from receipts without human interaction and use it in your backend applications for many accounting purposes.

&nbsp;

If you want to get more details about how to use the mindee python library:

<ul>
	<li>1.&nbsp;<a href="https://mindee.com/documentation/apis/tutorial-parsing-receipts-with-mindees-api/python/getting-started">Getting started with receipt parsing in python</a></li>
	<li>2.&nbsp;<a href="https://mindee.com/documentation/apis/tutorial-parsing-receipts-with-mindees-api/python/receipt-fields">Extracted receipt fields</a></li>
	<li>3.&nbsp;<a href="https://mindee.com/documentation/apis/tutorial-parsing-receipts-with-mindees-api/python/save-and-restore">Save and restore Receipt objects</a></li>
	<li>4.&nbsp;<a href="https://mindee.com/documentation/apis/tutorial-parsing-receipts-with-mindees-api/python/receipt-data-validation">Receipt data validations</a></li>
	<li>5.&nbsp;<a href="https://mindee.com/documentation/apis/tutorial-parsing-receipts-with-mindees-api/python/performances-benchmark">Benchmark Receipt OCR performances</a></li>
</ul>

&nbsp;

<h2>Tutorial</h2>

&nbsp;

The final script can be used within any python middleware application or REST API serving your frontend interfaces.

&nbsp;

In the end of this tutorial, we&rsquo;ll show you how to highlight the extracted receipt fields on the image using the open source computer vision library OpenCV.

&nbsp;

The final result looks like this:

&nbsp;

<img alt="" src="https://mindee-public-website-staging.s3.amazonaws.com/blog/2020/09/03/ezgifcom-video-to-gif-14.gif" />

&nbsp;

&nbsp;

<h2>API Prerequisites</h2>

&nbsp;

<ol>
	<li>You&rsquo;ll need a free Mindee account. <a href="https://platform.mindee.net/signup" target="_blank">Sign up</a>&nbsp;and confirm your email to login.</li>
	<li>A receipt.&nbsp; Look in your bag/wallet for a recent one, download one&nbsp;<a href="https://mindee-public-website-dev.s3.amazonaws.com/blog/2020/11/04/sample_receipt-35.jpg">here</a> &nbsp;or do a <a href="https://www.google.com/search?q=cafe+receipt" target="_blank">Google Image search</a> for a receipt and download a few to test with.</li>
</ol>

&nbsp;

<h2>&nbsp;</h2>

<h2>Setup the project</h2>

&nbsp;

Create an empty directory wherever you want in your laptop, we&rsquo;ll call ours &ldquo;python_receipt&rsquo;.

 
We recommend you to create a virtual environment before going to the next step, in order to avoid installing the required packages globally on your machine.To do so, run the following command:

&nbsp;

(macOS and Linux)

<pre>
<code class="language-bash">python -m venv venv</code></pre>

 
(windows)

<pre>
<code class="language-bash">py -m venv venv</code></pre>

&nbsp;

Then, you&#39;ll need to activate your venv, so that pip will install packages in that folder.

&nbsp;

(macOS and Linux)

<pre>
<code class="language-bash">source venv/bin/activate</code></pre>

 
(Windows)

<pre>
<code class="language-bash">.\env\Scripts\activate</code></pre>

&nbsp;

&nbsp;

We need now to Install mindee python library&nbsp;from PyPi using pip, a package manager for Python.

&nbsp;

<pre>
<code class="language-bash">pip install mindee</code></pre>

&nbsp;

If the install throws an error, you might need to upgrade your version of pip. To do so, run:

&nbsp;

<pre>
<code class="language-bash">pip install --upgrade pip </code></pre>

&nbsp;

Don&#39;t have pip installed? Try installing it, by running this from the command line:

&nbsp;

<pre>
<code class="language-bash">$ curl https://bootstrap.pypa.io/get-pip.py | python</code></pre>

&nbsp;

That&rsquo;s it for the setup, let&rsquo;s call the API.

&nbsp;

&nbsp;

<h2>Call the Mindee receipt API</h2>

&nbsp;

Login to the <a href="https://platform.mindee.net">platform</a> and click on the following item to access the Expense receipts API:

&nbsp;

<img alt="Expense receipt API card" src="https://mindee-public-website-staging.s3.amazonaws.com/blog/2020/09/03/expense_receipt_card.jpg" style="height:280px; width:300px" />

&nbsp;

If you don&rsquo;t have an API token yet for this API, go to the &ldquo;credentials&rdquo; section and create a new API Key.&nbsp;

&nbsp;

<img alt="new api token" src="https://mindee-public-website-staging.s3.amazonaws.com/blog/2020/09/03/new_token.jpg" style="height:146px; width:300px" />

&nbsp;

Create a &ldquo;main.py&rdquo; file in your project directory, and paste the following sample code:

&nbsp;

<pre>
<code class="language-python">from mindee import Client

filepath = "/path/to/my/file"
expense_receipt_token = "my-token-here"

mindee_client = Client(expense_receipt_token=expense_receipt_token)


if __name__ == "__main__":
 
 response = mindee_client.parse_receipt(filepath)

 print(response.receipt)</code></pre>

&nbsp;

&nbsp;

Replace the &ldquo;path/to/my/file&rdquo; placeholder in the code with the path of the receipt image you want to extract data from.

&nbsp;

Replace the &ldquo;my-token-here&rdquo; with the API token you created previously in the platform.

&nbsp;

You can get back to your console and run the script

&nbsp;

<pre>
<code class="language-python">python main.py</code></pre>

&nbsp;

You should see the details of the receipt printed in your console:

&nbsp;

<pre>
<code class="language-bash">-----Receipt data-----
Filename: receipt_.jpg
Total amount: 10.2 
Date: 2016-02-26
Category: food
Time: 15:20
Merchant name: CLACHAN
Taxes: 1.7; 20.0%; _
Total taxes: 1.7
----------------------</code></pre>

&nbsp;

Now let&rsquo;s take a look at the Receipt object.

&nbsp;

<h2>&nbsp;</h2>

<h2>Extracted receipt data</h2>

&nbsp;

The API extracts a list of different fields in the receipt (total amount, taxes, date &hellip;).&nbsp;

&nbsp;

The Receipt&nbsp;object under the&nbsp;<code>Response.receipt</code>&nbsp;attribute has all the features you need to get the receipt data and coordinates in the image to highlight the results.

&nbsp;

Here is a piece of code to show you how to get some of them:

&nbsp;

<pre>
<code class="language-python">from mindee import Client

filepath = "/path/to/my/file"
expense_receipt_token = "my-token-here"

mindee_client = Client(expense_receipt_token=expense_receipt_token)


if __name__ == "__main__":
 
 response = mindee_client.parse_receipt(filepath)

 # To get the total amount including taxes value (float), ex: 14.24
 print(response.receipt.total_incl.value)
 
 # To get the total amount excluding taxes value (float), ex: 10.21
 print(response.receipt.total_excl.value)
 
 # To get the total tax amount value (float), ex: 8.42
 print(response.receipt.total_tax.value)
 
 # To get the list of taxes
 print(response.receipt.taxes)

 # Loop on each Tax field
 for tax in response.receipt.taxes:
 # To get the tax amount
 print(tax.value)
 
 # To get the tax code for from a tax object
 print(tax.code)
 
 # To get the tax rate
 print(tax.rate)
 
 # To get the receipt date (string)
 print(response.receipt.date.value)
 
 # To get the receipt merchant name (string)
 print(response.receipt.merchant_name.value)
 
 # To get the receipt time (string)
 print(response.receipt.time.value)</code></pre>

&nbsp;

Run the script and check out the results in your console!

&nbsp;&nbsp;

Now we are going to add a piece of code that will highlight the features extracted on the pdf. It can help you or your users to very quickly validate the data extraction.

&nbsp;

&nbsp;

<h2>Highlight features on the image</h2>

&nbsp;

Let&rsquo;s try to highlight the features as if someone did it with a pen.

&nbsp;

First, you&rsquo;ll need to install the computer vision python library OpenCV if you don&rsquo;t have it already installed in your env. To do so, run:

&nbsp;

&nbsp;

<pre>
<code>pip install opencv-python</code></pre>

&nbsp;

Each field extracted in the&nbsp;<code>Response.receipt</code>&nbsp;object has a&nbsp;<code>Field.bbox</code>&nbsp;object. It&#39;s as list of vertices in relative coordinates representing the bounding box (% of the image width and height).

&nbsp;

Check&nbsp;what&#39;s inside for the total_incl.bbox for example, doing:

&nbsp;

<pre>
<code class="language-python">print(response.receipt.total_incl.bbox)</code></pre>

&nbsp;

Now that we know how to access the bounding boxes coordinates, we are going to create our&nbsp;highlighter function in which we can feed any field bounding boxes&nbsp;in different steps, as follows:

&nbsp;

<ol>
	<li>Check if the file is a pdf or an image and load it as an openCV image in both scenarios</li>
	<li>Create a mask image</li>
	<li>Loop on each feature coordinates and draw the feature rectangle on our mask</li>
	<li>Overlay the mask and original image with alpha</li>
	<li>Display image to the user</li>
</ol>

&nbsp;

Note: each coordinate returned by the API is relative (in % of the image). You&#39;ll see there is a relative to absolute conversion in the code.

&nbsp;

Here is the code step by step:

&nbsp;

<pre>
<code>def highlight_features(img_path, coordinates):
 # step 1: Loads the image as an openCV image
 cv_image = cv2.imread(img_path)

 # step 2: create mask image
 overlay = cv_image.copy()
 h, w = cv_image.shape[:2]

 # step 3: Loop on each feature coordinates and draw the feature rectangle on our mask
 for coord in coordinates:
 if len(coord):
 # Create points absolute coordinates
 pt1 = (int(w*coord[0][0]), int(h*coord[0][1]))
 pt2 = (int(w*coord[2][0]), int(h*coord[2][1]))
 cv2.rectangle(overlay, pt1, pt2, (70, 230, 244), cv2.FILLED)

 # step 4: Overlay the mask and original image with alpha
 final_image = cv2.addWeighted(overlay, 0.5, cv_image, 0.5, 0)

 # step 5: Display image to the user
 cv2.imshow("highlighted_image", cv2.resize(final_image, (600, int(600*h/w))))
 cv2.waitKey(0)</code></pre>

 
&nbsp;

Finally, we just have to modify a bit the code for&nbsp;executing the highlighting function and printing the results. Here is&nbsp;the final code:

&nbsp;

&nbsp;

<pre>
<code class="language-python">from mindee import Client
import cv2


def highlight_features(img_path, coordinates):
 # step 1: Load the file as a numpy image 
 cv_image = cv2.imread(img_path)

 # step 2: create mask image
 overlay = cv_image.copy()
 h, w = cv_image.shape[:2]

 # step 3: Loop on each feature coordinates and draw the feature rectangle on our mask
 for coord in coordinates:
 if len(coord):
 pt1 = (int(w*coord[0][0]), int(h*coord[0][1]))
 pt2 = (int(w*coord[2][0]), int(h*coord[2][1]))
 cv2.rectangle(overlay, pt1, pt2, (70, 230, 244), cv2.FILLED)

 # step 4: Overlay the mask and original image with alpha
 final_image = cv2.addWeighted(overlay, 0.5, cv_image, 0.5, 0)

 # step 5: Display image to the user
 cv2.imshow("highlighted_image", cv2.resize(final_image, (600, int(600*h/w))))
 cv2.waitKey(0)


filepath = "/path/to/my/file"
expense_receipt_token = "my-token-here"

mindee_client = Client(expense_receipt_token=expense_receipt_token)


if __name__ == "__main__":
 
 response = mindee_client.parse_receipt(filepath)

 print(response.receipt)

 highlight_features(
 filepath,
 [
 response.receipt.total_incl.bbox,
 response.receipt.time.bbox,
 response.receipt.merchant_name.bbox,
 response.receipt.date.bbox
 ]
 )
</code></pre>

 
&nbsp;

And the final result!

&nbsp;

<img alt="" src="https://mindee-public-website.s3.amazonaws.com/blog/2020/11/05/highlghted_image.jpg" />

&nbsp;

<h2>Conclusion</h2>

&nbsp;

In just over 1 second, a receipt was uploaded, the API extracted the receipt data, then the result was&nbsp;parsed&nbsp;and&nbsp;highlighted in&nbsp;the image. Cool right?

&nbsp;

If you wanted to use this kind of script to display results to your users, i&#39;d advice you to do the highlighting in the front-end application, as sending images back from your middleware is not the best option because of payload sizes. The other solution would be to store the final image using cv2.imwrite(...) but it would make your client download the result. To help you with client-side rendering for your images, you can check out our&nbsp;<a href="https://github.com/publicMindee/react-mindee-js">open source ReactJS SDK</a>.

&nbsp;

If you have questions, please reach out to us in the chat widget in the bottom right.

This tutorial explains how to extract data from receipt image using python and Mindee"s API, then how to parse the result and highlight the different features on the source image.

<h1>Getting started</h1>

<h2>&nbsp;</h2>

&nbsp;

Get started with receipt parsing in python.

&nbsp;

<h2>1. Install the mindee python helper library</h2>

&nbsp;

Install from PyPi using&nbsp;<a href="https://pip.pypa.io/en/latest/" rel="nofollow">pip</a>, a package manager for Python.

&nbsp;

<pre>
<code class="language-bash">pip install mindee</code></pre>

&nbsp;

Don&#39;t have pip installed? Try installing it, by running this from the command line:

&nbsp;

<pre>
<code class="language-bash">$ curl https://bootstrap.pypa.io/get-pip.py | python</code></pre>

&nbsp;

Getting started with the Mindee API couldn&#39;t be easier. Create a&nbsp;<code>Client</code>&nbsp;and you&#39;re ready to go.

&nbsp;

&nbsp;

&nbsp;

<h2>2. Instantiate your Client</h2>

&nbsp;

The&nbsp;<code>mindee.Client</code>&nbsp;needs your API credentials. You can either pass these directly to the constructor (see the code below) or via environment variables.

&nbsp;

Depending on what type of document you want to parse, you need to add specifics auth token for each endpoint.

&nbsp;

<pre>
<code class="language-python">from mindee import Client

mindee_client = Client(
 expense_receipt_token="your_expense_receipts_api_token_here",
 raise_on_error=True
)</code></pre>

&nbsp;

<ul>
	<li>expense_receipt_token:&nbsp;(string) API key for expense_receipts endpoint</li>
	<li>raise_on_error: (boolean, default True) Specify wheter or not raising an Exception when HTTP errors occur. If set to False, when calling the parse_receipt method, the <code>Response.receipt</code> object returned will have be set to None and the&nbsp; <code>Response.receipts</code>&nbsp;list will be empty.</li>
</ul>

&nbsp;

&nbsp;

We suggest storing your credentials as environment variables. Why? You&#39;ll never have to worry about committing your credentials and accidentally posting them somewhere public.

&nbsp;

&nbsp;

&nbsp;

<h2>3. Parse a receipt</h2>

&nbsp;

Using the parse_receipt&nbsp;method from your client object, you can pass any image or pdf file to get the receipt data.

&nbsp;

<pre>
<code class="language-python">from mindee import Client

mindee_client = Client(
 expense_receipt_token="your_expense_receipts_api_token_here",
 raise_on_error=True
)

parsed_data = mindee_client.parse_receipt("/path/to/file")
</code></pre>

&nbsp;

&nbsp;

<h3>Input types</h3>

&nbsp;

You can pass your input file in three ways:

&nbsp;

From file path

<pre>
<code class="language-python">receipt_data = mindee_client.parse_receipt('/path/to/file', input_type="path")</code></pre>

&nbsp;

&nbsp;From a file object

<pre>
<code class="language-python">with open('/path/to/file', 'rb') as fp:
 receipt_data = mindee_client.parse_receipt(fp, input_type="file")
</code></pre>

&nbsp;

From a base64

<pre>
<code class="language-python">receipt_data = mindee_client.parse_receipt(base64_string, input_type="base64")</code></pre>

&nbsp;

&nbsp;

receipt_data structure

&nbsp;

The receipt_data object returned by the&nbsp;<code>parse_receipt</code>&nbsp;method contains the following elements:

&nbsp;

receipt_data.receipt

&nbsp;

The receipt attribute&nbsp;is the Receipt object constructed by gathering all the pages into a single document. The method used for creating a single receipt object with multiple pages relies on fields confidence scores. Basically, we iterate over each page and for each field, we keep the one that has the highest probability.

&nbsp;

<pre>
<code class="language-python">receipt_data.receipt # returns a unique object from class Receipt</code></pre>

<h4>&nbsp;</h4>

&nbsp;

receipt_data.receipts

&nbsp;

For multi pages pdf, the receipts attribute contains one receipt object per input. For a single-page pdf or an image, the list will only contain 1 item, and for multi-page pdf, it will contain one item per page.

&nbsp;

<pre>
<code class="language-python">receipt_data.receipts # [Receipt, Receipt ...] </code></pre>

<h4>&nbsp;</h4>

&nbsp;

receipt_data.http_response

&nbsp;

Contains the full Mindee API HTTP response object in JSON format

&nbsp;

<pre>
<code class="language-python">receipt_data.http_response # full HTTP request object </code></pre>

&nbsp;

&nbsp;

&nbsp;

<h2>4. Display the results</h2>

&nbsp;

You only have to print your receipt object to display the different extracted fields:

&nbsp;

<pre>
<code class="language-python">from mindee import Client

mindee_client = Client(
 expense_receipt_token="your_expense_receipts_api_token_here",
 raise_on_error=True
)

receipt_data = mindee_client.parse_receipt("/path/to/receipt")

print(receipt_data.receipt)</code></pre>

&nbsp;

&nbsp;

Getting started with Mindee receipt OCR for parsing receipt image using python.

<h1>Receipt fields</h1>

&nbsp;

&nbsp;

<h2>General fields structure</h2>

&nbsp;

Each receipt object contains a set of different fields.

&nbsp;

To access a receipt object, you need to create a <code>mindee.Client&nbsp;</code>and call the <code>Client.parse_receipt</code>&nbsp;method:

&nbsp;

<pre>
<code>from mindee import Client

# Instantiate your client
mindee_client = Client(expense_receipts_token="your_token_here")

# Call the Mindee expense receipts endpoint and parse the result
receipt_data = mindee_client.parse_receipt("/path/to/my/file")</code></pre>

&nbsp;

Whatever the field type, each of them contains the four following attributes:

&nbsp;

<ul>
	<li>Value: (Str or Float depending on the field type), corresponds to the field value. Set to None if the field was not extracted.</li>
	<li>Probability: (Float), confidence score of the field prediction.</li>
	<li>bbox:(Array[Float]), contains the relative vertices coordinates of the bounding box containing the field in the image. If the field is not written, the bbox is an empty array.&nbsp;</li>
	<li>reconstructed: (Bool), True if the field was reconstructed using other fields.</li>
</ul>

&nbsp;

Depending on the Field type, there can be extra attributes.

&nbsp;

&nbsp;

&nbsp;

<h2>Total amounts</h2>

&nbsp;

&nbsp;receipt.total_incl:&nbsp;Total amount including taxes

<pre>
<code class="language-python"># To get the total amount including taxes value (float), ex: 14.24
total_incl = receipt_data.receipt.total_incl.value
</code></pre>

&nbsp;

&nbsp;receipt.total_excl:&nbsp;Total amount excluding taxes

<pre>
<code class="language-python"># To get the total amount excluding taxes value (float), ex: 10.21
total_excl = receipt_data.receipt.total_excl.value</code></pre>

&nbsp;

receipt.total_tax: Total tax value reconstructed from tax lines

<pre>
<code class="language-python"># To get the total tax amount value (float), ex: 8.42
total_tax = receipt_data.receipt.total_tax.value</code></pre>

<img src="https://mindee-public-website-dev.s3.amazonaws.com/blog/2020/10/26/image-20201026171243-1.gif" />

&nbsp;

&nbsp;

<h2>Taxes</h2>

&nbsp;

receipt.taxes:&nbsp;Array of Tax fields

Each of the tax field has two extra attributes:

&nbsp;

<ul>
	<li>code: (String), Optional tax code. (HST, GST... for canadian; City Tax, State tax for US etc..)</li>
	<li>rate: (Float), Optional tax rate.</li>
</ul>

&nbsp;

<pre>
<code class="language-python"># To get the list of taxes
taxes = receipt_data.receipt.taxes

# Loop on each Tax field
for tax in taxes:
 # To get the tax amount
 tax_amount = tax.value

 # To get the tax code for from a tax object
 tax_code = tax.code
 
 # To get the tax rate
 tax_rate = tax.rate</code></pre>

&nbsp;

&nbsp;

&nbsp;

<h2>Dates</h2>

&nbsp;

Each date field comes with an extra attribute:

&nbsp;

<ul>
	<li>date_object: (Datetime), datetime object from python datetime.date library</li>
</ul>

&nbsp;

&nbsp;

&nbsp;receipt.date: Payment date related to the receipt

<pre>
<code class="language-python"># To get the receipt date of issuance (string)
receipt_date = receipt_data.receipt.value</code></pre>

&nbsp;

&nbsp;

<h2>Supplier informations</h2>

&nbsp;

&nbsp;receipt.supplier:&nbsp;Supplier name as written in the receipt&nbsp;(logo)

<pre>
<code class="language-python"># To get the supplier name
supplier_name = receipt_data.receipt.supplier.value</code></pre>

&nbsp;

&nbsp;

&nbsp;

<h2>Locale and currency</h2>

&nbsp;

locale: Language ISO code

Contains an extra attribute:

&nbsp;

<ul>
	<li>currency: (String), ISO currency code</li>
</ul>

&nbsp;

<pre>
<code class="language-python"># To get the total language code
language = receipt_data.receipt.locale.value

# To get the receipt currency code
currency = receipt_data.receipt.locale.currency</code></pre>

&nbsp;

&nbsp;

<h2>&nbsp;Category</h2>

&nbsp;

category: Receipt category among the list:&nbsp; toll, food, parking, transport, accommodation, gasoline, miscellaneous

<pre>
<code class="language-python"># To get the category
category = receipt_data.receipt.category.value</code></pre>

&nbsp;

&nbsp;

<h2>&nbsp;Time</h2>

&nbsp;

time: Time of purchase with 24 hours formatting (hh:mm)

<pre>
<code class="language-python"># To get the category
category = receipt_data.receipt.time.value</code></pre>

&nbsp;

&nbsp;

How to use extracted receipt data with Mindee python helper library.

Receipt fields

<h1>Save and restore</h1>

&nbsp;

The&nbsp;<code>mindee.Client</code>&nbsp;includes methods for saving and restoring api calls. Those methods are helpful if you want to perform tests or if you simply want to reload receipts objects corresponding to your files without performing any additional request.

&nbsp;

&nbsp;

<h2>Save a call</h2>

&nbsp;

The <code>mindee.Response</code> object returned by the <code>mindee.Client</code> when parsing a file is responsible for wrapping all the data from Mindee API calls. It&nbsp;includes a <code>dump</code> method that can be used as follows:

&nbsp;

<pre>
<code class="language-python">from mindee import Client

mindee_client = Client(expense_receipts_token="your_expense_receipts_api_token_here")

receipt_data = mindee_client.parse_receipt("/path/to/your/file", input_type="file")

receipt_data.dump("./receipt_dump.json")</code></pre>

&nbsp;

&nbsp;

<h2>Restore a call</h2>

&nbsp;

To restore a client response object, you can use the <code>mindee.Response</code> as follows:

&nbsp;

<pre>
<code class="language-python">from mindee import Response

receipt_data = Response.load("./receipt_dump.json")</code></pre>

&nbsp;

&nbsp;

Save and restore Mindee expense receipt API response with Mindee python helper library

<h1>Validation checks</h1>

&nbsp;

&nbsp;

Each receipt object comes with a set of data validation controls. it can be accessed under the attribute <code>Receipt.checklist</code>.

&nbsp;

Example:

&nbsp;

<pre>
<code class="language-python">from mindee import Client

mindee_client = Client(
 expense_receipts_token="your_expense_receipts_api_token_here",
 raise_on_error=True
)

parsed_data = mindee_client.parse_receipt("/path/to/file")

print(parsed_data.receipt.checklist)</code></pre>

&nbsp;

&nbsp;

This sample code should print something like:

&nbsp;

<pre>
<code class="language-bash">{'taxes_match_total_incl': True}</code></pre>

&nbsp;

&nbsp;

taxes_match_total_incl: Try to reconstruct the total amount including taxes with taxes line details&nbsp;and compare the result against the extracted total_incl field.

&nbsp;

<h2>Interpretation: <code>True</code> checks</h2>

&nbsp;

Mindee&#39;s algorithms extract all fields independantly, whatever the type of document. For example, they don&#39;t use the&nbsp;extracted taxes&nbsp;to reconstruct the total amounts. This way, when trying to reconstruct some fields from others with post-processing and comparing them to raw extracted fields, it&#39;s very unlikely to get a matching reconstruction although the predictions were wrong.

&nbsp;

&nbsp;

<pre>
<code class="language-python">from mindee import Client


mindee_client = Client(
 expense_receipts_token="your_expense_receipts_api_token_here",
 raise_on_error=True
)

receipt_data = mindee_client.parse_receipt("path/to/file")

if receipt_data.receipt.checklist["taxes_match_total_incl"] is True:
 # We are sure that taxes and total_incl fields were extracted correctly
 pass
</code></pre>

&nbsp;

&nbsp;

The&nbsp;<code>mindee</code>&nbsp;library will enforce probabilities to 1 for each field involved in validated check.

&nbsp;

Built-in receipt data validation rules with Mindee python helper library.

Receipt data validation

<h1>Performances benchmark</h1>

&nbsp;

In a few steps, you can use the mindee python library to perform benchmarks of receipt OCR API perfomances against your test dataset.

&nbsp;

<ol>
	<li>Perform api calls for each file in your test set</li>
	<li>Create <code>mindee.Receipt</code> objects from your data (csv, json or whatever)</li>
	<li>Perform the benchmark and analyze the results</li>
</ol>

&nbsp;

&nbsp;

&nbsp;

<h2>1. Perform api calls for each file in your test set</h2>

&nbsp;

Before running the benchmark, we need to collect all the receipt predictions from Mindee API and store them so we can perform the benchmark later without calling the API again.

&nbsp;

To do so, assuming that all your files are stored into a &quot;./test_dataset&quot; folder and that we want to save all the responses into a &quot;./mindee_responses&quot; folder, here is the code:

&nbsp;

<pre>
<code class="language-python">from mindee import Client
import os


TEST_DATA_PATH = "./test_dataset"
RESPONSES_PATH = "./mindee_responses"


mindee_client = Client(expense_receipts_token="your_expense_receipts_api_token_here")


# Loop over all your test files
for test_filename in os.listdir(TEST_DATA_PATH):
 
 # Get the current file path
 test_file_path = os.path.join(TEST_DATA_PATH, test_filename)
 
 # To make sure we don't stop the process if an error occurs
 try:
 # Parse the current file
 mindee_response = mindee_client.parse_receipt(test_file_path)
 
 # Store the response inside json file to be restored later
 # In this example we use the test file name in the json filename to 
 # be able to retrieve the corresponding file
 response_filepath = os.path.join(RESPONSES_PATH, test_filename+".json")
 mindee_response.dump(response_filepath)

 except Exception as e:
 # In case of error, print the filename so you can understand later
 # what happened
 print(test_filename, e)
</code></pre>

&nbsp;

&nbsp;

&nbsp;

<h2>2. Create <code>mindee.Receipt</code> objects from your data (csv, json,&nbsp;whatever...)</h2>

&nbsp;

The <code>mindee.Receipt</code> class contains a static&nbsp;<code>compare()</code> method that takes as inputs two <code>mindee.Receipt</code> objects. Before running our final script, we need now to create a&nbsp;<code>mindee.Receipt object</code> containing the true labels for each fields.

&nbsp;

We&#39;ll use a csv file in this example and the&nbsp;<a href="https://pandas.pydata.org/">pandas</a>&nbsp;library.

&nbsp;

<h3>ground_truth.csv</h3>

<pre>
<code class="language-bash">filename,total_incl,date,taxes
1.jpg,149.5,2020-12-01,11.5-10|23-20</code></pre>

<h3>&nbsp;</h3>

To construct a Receipt&nbsp;object from this dummy csv example, you can simply do:

&nbsp;

<pre>
<code class="language-python">import pandas as pd
from mindee import Receipt

ground_truth_df = pd.read_csv("./ground_truth.csv")


def receipt_from_csv_row(df_row):
 taxes_list = df_row["taxes"].split("|")
 return Receipt(
 total_incl=df_row["total_incl"],
 date=df_row["date"],
 taxes=[(t.split("-")[0], t.split("-")[1]) for t in taxes_list]
 )


for index, df_row in ground_truth_df.iterrows():
 receipt_truth = receipt_from_csv_row(df_row)
 print(receipt_truth)
</code></pre>

&nbsp;

&nbsp;

Running this code should print receipt data from your csv file in your console.

&nbsp;

&nbsp;

&nbsp;

<h2>3.&nbsp;Perform the benchmark and see the results</h2>

&nbsp;

Last step, now we need to wrap it all up.

&nbsp;

The Benchmark class has two methods <code>Benchmark.add</code>&nbsp;and <code>Benchmark.save</code>&nbsp;for adding comparison between two receipt objects, and saving the final metrics.

&nbsp;

&nbsp;

<pre>
<code class="language-python">import pandas as pd
from mindee import Response, Receipt, Benchmark
import os

TEST_DATA_PATH = "./test_dataset"
RESPONSES_PATH = "./mindee_responses"
BENCHMARK_PATH = "./benchmark"

ground_truth_df = pd.read_csv("./ground_truth.csv")
benchmark = Benchmark(BENCHMARK_PATH)


def receipt_from_csv_row(df_row):
 # Your method for creating the Receipt ground truth
 pass
 

# Loop over each file in our csv
for index, df_row in ground_truth_df.iterrows():
 try:
 # Get test file path
 test_file_path = os.path.join(TEST_DATA_PATH, df_row["filename"])

 # Create ground truth receipt object
 ground_truth_receipt = receipt_from_csv_row(df_row)
 
 # Load the mindee Response for the current file
 mindee_response = Response.load(os.path.join(RESPONSES_PATH, df_row["filename"] + ".json"))

 # Add the comparison between the two receipts to the benchmark
 benchmark.add(
 Receipt.compare(mindee_response.receipt, ground_truth=ground_truth_receipt),
 df_row["filename"]
 )
 
 except Exception as e:
 print(df_row["filename"], e)

benchmark.save()</code></pre>

&nbsp;

&nbsp;

Inside our benchmark folder, you should see a new directory was created, and a metrics.png file shoud have been created inside with the different metrics:

&nbsp;

<img alt="" src="https://mindee-public-website.s3.amazonaws.com/blog/2020/11/05/metrics-1.png" />

&nbsp;

The receipt benchmark runs on the 7 fields as shown above. For each of them, you get an information of:

accuracy: the proportion of correct predictions

precision: The proportion of correct predictions among all the non null predictions

Benchmark Receipt OCR performances using Mindee python helper library

<h1>Extract receipt data with Mindee&rsquo;s API using reactjs</h1>

&nbsp;

In this tutorial, you&#39;ll learn how to create an optimized and interactive user interface for automatic receipt data extraction using ReactJS and Mindee. We are going to use both Mindee API and open source <a href="https://github.com/publicMindee/react-mindee-js">ReactJS SDK</a>. The SDK has a lot of features built-in such as shapes visualisation and interactions, zoom, multi pages ... It will help&nbsp;you create any dynamic user experience based on the receipt image&nbsp;to help your users validate very quickly the results predicted by the API.

&nbsp;

In 10 minutes,&nbsp;you&#39;ll get something like this:

&nbsp;

<img alt="Mindee receipt parsing react js" src="https://mindee-public-website-staging.s3.amazonaws.com/blog/2020/09/04/ezgifcom-optimize-2.gif" />

&nbsp;

Let&#39;s get started!

&nbsp;

<h2>&nbsp;</h2>

<h2>API Prerequisites</h2>

&nbsp;

<ol>
	<li>We are using create-react-app framework for bootstrapping the app. If you don&#39;t have it installed, you can do it&nbsp;<a href="https://create-react-app.dev/docs/getting-started/">here</a>. You can also use another React framework for creating the app but some code may differ from what is written in this tutorial.</li>
	<li>You&rsquo;ll need a free Mindee account. <a href="https://platform.mindee.net/signup" target="_blank">Sign up</a>&nbsp;and confirm your email to login.</li>
	<li>A receipt.&nbsp; Look in your bag/wallet for a recent one, or do a <a href="https://www.google.com/search?q=cafe+receipt" target="_blank">Google Image search</a> for a receipt and download a few to test with.</li>
</ol>

&nbsp;

&nbsp;

<h2>Setup your project</h2>

&nbsp;

First, we need to create a new react application using create-react-app. Open up your terminal and go to the directory you want to create your project in, and create your app with the name you want (we call ours receipt_parsing):

&nbsp;

<pre>
<code class="language-bash">npx create-react-app receipt_parsing</code></pre>

&nbsp;

To make sure everything went well, go to your directory and run the app

&nbsp;

<pre>
<code class="language-bash">cd receipt_parsing
yarn start</code></pre>

&nbsp;

Or using npm

&nbsp;

<pre>
<code class="language-bash">cd receipt_parsing
npm run start</code></pre>

&nbsp;

Your browser is going to open a new window and run the app.

&nbsp;

A last package install before we can start coding! As we are going to use Mindee ReactJS SDK, we need to install it. It&#39;s open-source and you can find all the documentation on the <a href="https://github.com/publicMindee/react-mindee-js">Github repo</a>. To install in your React app, just run in your terminal:

&nbsp;

<pre>
<code class="language-bash">yarn add react-mindee-js</code></pre>

&nbsp;

Or using npm:

&nbsp;

<pre>
<code class="language-bash">npm install react-mindee-js</code></pre>

&nbsp;

Now, using your favourite IDE, open the App.js file that was created by create-react-app, inside the src folder of your project directory. In the whole tutorial, we are going to work only on this file. Once you access the App.js file in your terminal, delete everything and just paste this code:

&nbsp;

<pre>
<code class="language-ruby">import React, {useState} from 'react';

import {
 AnnotationViewer,
 createPolygonFromCoordinates
} from "react-mindee-js";

const HelloWorld = () =&gt; {
 const [image, setImage] = useState(null)

 const shapes = [
 {
 polygon: createPolygonFromCoordinates([[0.66, 0.3], [0.12, 0.5], [0.52, 0.8], [0.82, 0.24]]),
 color: "#fd3246",
 data:"This is red shape",
 index: 0,
 active: {
 color: '#00f0f0',
 lineWidth: 3
 }
 }
 ]

 return (
 &lt;div&gt;
 &lt;input type="file" onChange={e =&gt; setImage(URL.createObjectURL(e.target.files[0]))}/&gt;
 {
 image &amp;&amp; &lt;div style={{ height: '500px', width: '400px'}}&gt;
 &lt;AnnotationViewer
 shapes={shapes}
 image={image}
 onShapeClick={shape =&gt; console.log('shapeClicked', shape)}
 onShapeHover={shape =&gt; console.log('shapeHovered', shape)}
 /&gt;
 &lt;/div&gt;
 }
 &lt;/div&gt;
 );
}
export default HelloWorld;</code></pre>

&nbsp;

We define a polygon by its [x,y] coordinates and colors.&nbsp; When an image is added from the form, we call the onChange to apply the polygon, and write to the console on Click and Hover actions.

You&#39;re all setup to start. When you run the app, you should see a&nbsp;file input button on your web page, and once you upload an image using it, you get something like:

&nbsp;

<img alt="" src="https://mindee-public-website-staging.s3.amazonaws.com/blog/2020/09/04/hello_world_receipt_react.jpg" />

The ploygon is defined but the four [x,y] vertices, and the color is red, but on hover, it turns blue (00f0f0).

<h2>&nbsp;</h2>

<h2>&nbsp;</h2>

<h2>Parse the receipt</h2>

&nbsp;

Log&nbsp;onto the <a href="https://platform.mindee.net/login">platform</a> and get into your Expense receipt API environment by clicking the following card.

<img alt="Expense receipt API card" src="https://mindee-public-website-staging.s3.amazonaws.com/blog/2020/09/03/expense_receipt_card.jpg" />

&nbsp;

If you don&rsquo;t have an API token yet for this API, go to the &ldquo;credentials&rdquo; section and create a new API Key.&nbsp;

&nbsp;

<img alt="new api token" src="https://mindee-public-website-staging.s3.amazonaws.com/blog/2020/09/03/new_token.jpg" />

&nbsp;

Inside your App.js file, replace everything with this code, and replace the &#39;my-token-here&#39; placeholder with the one you just created on the platform:

&nbsp;

<pre>
<code class="language-javascript">import React, {useState} from 'react';
import {
 AnnotationViewer,
 formatPrediction
} from "react-mindee-js";

const HelloWorld = () =&gt; {
 const [image, setImage] = useState(null)
 const [shapes, setShapes] = useState([])

 const parseReceipt = (e) =&gt; {
 let myFile = e.target.files[0]

 if (!myFile) {return}
 
 setImage(URL.createObjectURL(myFile))

 let data = new FormData();
 data.append("file", myFile, myFile.name);

 let xhr = new XMLHttpRequest();

 xhr.addEventListener("readystatechange", function () {
 if (this.readyState === 4) {
 let json_response = JSON.parse(this.responseText)
 setShapes(formatPrediction(json_response.predictions[0]))
 }
 });

 xhr.open("POST", "https://api.mindee.net/products/expense_receipts/v2/predict");
 xhr.setRequestHeader("X-Inferuser-Token", "my-token-here");
 xhr.send(data);
 }

 return (
 &lt;div&gt;
 &lt;input type="file" onChange={e =&gt; parseReceipt(e)}/&gt;
 {
 image &amp;&amp; &lt;div style={{ height: '500px', width: '400px'}}&gt;
 &lt;AnnotationViewer
 shapes={shapes}
 image={image}
 onShapeClick={shape =&gt; console.log('shapeClicked', shape)}
 onShapeHover={shape =&gt; console.log('shapeHovered', shape)}
 /&gt;
 &lt;/div&gt;
 }
 &lt;/div&gt;
 );
}
export default HelloWorld;</code></pre>

&nbsp;

In this iteration, we have replaced the onChange callback function to&nbsp;a new function parseReceipt. Basically, this function will call the Receipt parsing API (and send the image to Mindee to extract the receipt details), and transform the results to very easily generate shapes using formatPrediction() function from the SDK.

&nbsp;

Run the app, you should see something like this when you upload a receipt image:

&nbsp;

<img alt="react mindee parse receipt" src="https://mindee-public-website-staging.s3.amazonaws.com/blog/2020/09/04/hello_world_receipt_react_step.jpg" />

&nbsp;

All boxes displayed on the image are interractive and you can bind custom onClick and onMouseOver mouse events functions to them. You can also try to zoom in the image, try to move the image inside the viewer etc.. All features are explained on the Github repo.

&nbsp;

Important note: Calling the receipt parsing API client-side is not&nbsp;secure, as your API token is present in the browser data, potentially exposing this to the world.&nbsp; This approach should only be used for&nbsp;prototyping,&nbsp;you&#39;ll need a back-end middleware to make a secure production app.

&nbsp;

Now we are just going to change a few lines of codes to display the data extracted by the API. To do so, we are going to add a new state object &#39;receiptData&#39;, and display a few features on the web page. In your code, change the HelloWorld component to this (don&#39;t forget to fix your api token):

&nbsp;

<pre>
<code class="language-javascript">const HelloWorld = () =&gt; {
 const [image, setImage] = useState(null)
 const [shapes, setShapes] = useState([])
 const [receiptData, setReceiptData] = useState(null)

 const parseReceipt = (e) =&gt; {
 let myFile = e.target.files[0]

 if (!myFile) {
 return
 }
 setImage(URL.createObjectURL(myFile))

 let data = new FormData();
 data.append("file", myFile, myFile.name);

 let xhr = new XMLHttpRequest();

 xhr.addEventListener("readystatechange", function () {
 if (this.readyState === 4) {
 let json_response = JSON.parse(this.responseText)
 setShapes(formatPrediction(json_response.predictions[0]))
 setReceiptData(json_response.predictions[0])
 }
 });

 xhr.open("POST", "https://api.mindee.net/products/expense_receipts/v2/predict");
 xhr.setRequestHeader("X-Inferuser-Token", "my-token-here");
 xhr.send(data);
 }

 return (
 &lt;div&gt;
 &lt;input type="file" onChange={e =&gt; parseReceipt(e)}/&gt;
 {
 image &amp;&amp; &lt;&gt;
 &lt;div style={{height: '500px', width: '400px'}}&gt;
 &lt;AnnotationViewer
 shapes={shapes}
 image={image}
 /&gt;
 &lt;/div&gt;
 {
 receiptData &amp;&amp;
 &lt;div style={{marginLeft: "50px"}}&gt;
 &lt;p&gt;Total amount : {receiptData.total.amount}&lt;/p&gt;
 &lt;p&gt;Date: {receiptData.date.iso}&lt;/p&gt;
 &lt;p&gt;Time: {receiptData.time.iso}&lt;/p&gt;
 &lt;p&gt;Category: {receiptData.category.value}&lt;/p&gt;
 &lt;/div&gt;
 }
 &lt;/&gt;
 }
 &lt;/div&gt;
 );
}</code></pre>

&nbsp;

Run the app, upload a receipt, you should get:

&nbsp;

<img alt="Reactjs mindee parsing receipt" src="https://mindee-public-website-staging.s3.amazonaws.com/blog/2020/09/04/hello_world_receipt_react_step.jpg" />

&nbsp;

We&#39;re almost done!

&nbsp;

<h2>Bind interractions between images boxes and corresponding receipt text features</h2>

&nbsp;

To make sure your user can validate very easily data extracted by the API, the ReactJS SDK allows you to add interactions with visual features on the images and the rest of your app.

&nbsp;

First, we&#39;ll create a function that change the isActive attribute inside a shape object, from it&#39;s name. The SDK will understand this attribute and will modify the shape layout (you can modify the active layout by changing colors and line width in the active attribute of a shape).

&nbsp;

<pre>
<code class="language-javascript">const highlightFeature = (featureName) =&gt; {
 let newShapes = []
 shapes.forEach(shape =&gt; {
 shape.isActive=(shape.featureName === featureName)
 newShapes.push(shape)
 }
 )
 setShapes(newShapes)
}</code></pre>

&nbsp;

We&#39;ll then modify a bit the rendered code, and bind on the onMouseOver event of the text features the highlight function.

&nbsp;

<pre>
<code class="language-javascript">return (
 &lt;div&gt;
 &lt;input type="file" onChange={e =&gt; parseReceipt(e)}/&gt;
 {
 image &amp;&amp; &lt;&gt;
 &lt;div style={{height: '500px', width: '400px'}}&gt;
 &lt;AnnotationViewer
 shapes={shapes}
 image={image}
 /&gt;
 &lt;/div&gt;
 {
 receiptData &amp;&amp;
 &lt;div style={{marginLeft: "50px"}}&gt;
 &lt;p onMouseOver={() =&gt; highlightFeature('total')}&gt;Total amount : {receiptData.total.amount}&lt;/p&gt;
 &lt;p onMouseOver={() =&gt; highlightFeature('date')}&gt;Date: {receiptData.date.iso}&lt;/p&gt;
 &lt;p onMouseOver={() =&gt; highlightFeature('time')}&gt;Time: {receiptData.time.iso}&lt;/p&gt;
 &lt;p&gt;Category: {receiptData.category.value}&lt;/p&gt;
 &lt;/div&gt;
 }
 &lt;/&gt;
 }
 &lt;/div&gt;
 );</code></pre>

&nbsp;

Final App.js

<pre>
<code class="language-javascript">import React, {useState} from 'react';
import './App.css';

import {
 AnnotationViewer,
 formatPrediction
} from "react-mindee-js";

const HelloWorld = () =&gt; {
 const [image, setImage] = useState(null)
 const [shapes, setShapes] = useState([])
 const [receiptData, setReceiptData] = useState(null)

 const parseReceipt = (e) =&gt; {
 let myFile = e.target.files[0]

 if (!myFile) {
 return
 }
 setImage(URL.createObjectURL(myFile))

 let data = new FormData();
 data.append("file", myFile, myFile.name);

 let xhr = new XMLHttpRequest();

 xhr.addEventListener("readystatechange", function () {
 if (this.readyState === 4) {
 let json_response = JSON.parse(this.responseText)
 setShapes(formatPrediction(json_response.predictions[0]))
 setReceiptData(json_response.predictions[0])
 }
 });

 xhr.open("POST", "https://api.mindee.net/products/expense_receipts/v2/predict");
 xhr.setRequestHeader("X-Inferuser-Token", "my-token-here");
 xhr.send(data);
 }

 const highlightFeature = (featureName) =&gt; {
 let newShapes = []
 shapes.forEach(shape =&gt; {
 shape.isActive=(shape.featureName === featureName)
 newShapes.push(shape)
 }
 )
 setShapes(newShapes)
 }

 return (
 &lt;div&gt;
 &lt;input type="file" onChange={e =&gt; parseReceipt(e)}/&gt;
 {
 image &amp;&amp; &lt;&gt;
 &lt;div style={{height: '500px', width: '400px'}}&gt;
 &lt;AnnotationViewer
 shapes={shapes}
 image={image}
 /&gt;
 &lt;/div&gt;
 {
 receiptData &amp;&amp;
 &lt;div style={{marginLeft: "50px"}}&gt;
 &lt;p onMouseOver={() =&gt; highlightFeature('total')}&gt;Total amount : {receiptData.total.amount}&lt;/p&gt;
 &lt;p onMouseOver={() =&gt; highlightFeature('date')}&gt;Date: {receiptData.date.iso}&lt;/p&gt;
 &lt;p onMouseOver={() =&gt; highlightFeature('time')}&gt;Time: {receiptData.time.iso}&lt;/p&gt;
 &lt;p&gt;Category: {receiptData.category.value}&lt;/p&gt;
 &lt;/div&gt;
 }
 &lt;/&gt;
 }
 &lt;/div&gt;
 );
}
export default HelloWorld;</code></pre>

&nbsp;

Running your app, you should see now that when you hover over a text line at the bottom,&nbsp;the color of the box highlighting the value on the receipt&nbsp;will change. You can modify the colours as you want to make the change more or less dramatic, as you please.&nbsp; All of the details for this is&nbsp;explained in the Github readme.

&nbsp;

<h2>Conclusion</h2>

&nbsp;

Using our ReactJS library makes identifying regions in the image easy.&nbsp; The Mindee Receipt Parsing API extracts the relevant details (and the location in the image). Then the react library highlights the locations with the AnnotationViewer.&nbsp;Combined, you have a powerful tool to extract data from images, and display them for verification to your users.

&nbsp;

&nbsp;

Tutorial on how to extract data from a receipt image through a reactjs user interface. You will learn how to parse the different data extracted by the Mindee API and how to create a beautiful user interface using Mindee ReactJS library.

reactjs

<h1>Extract receipt data with Mindee&rsquo;s API using NodeJS</h1>

&nbsp;

In&nbsp;this example, we will build from scratch, a NodeJS based web application that accepts a receipt image or PDF, and returns extracted data from the receipt.

&nbsp;

The code for the sample code is availble on <a href="https://github.com/dougsillars/mindee_node_receipt">Github</a>, and also available on <a href="https://glitch.com/edit/#!/grove-tree-sprite">Glitch</a>.&nbsp; For simplicity, we&#39;ll follow along in the Glitch example.

&nbsp;

<div class="youtube-embed-wrapper" style="height:0; overflow:hidden; padding-bottom:56.25%; padding-top:30px; position:relative"><iframe allowfullscreen="" frameborder="0" height="360" src="https://www.youtube.com/embed/3r5HhjShNtk" style="position:absolute;top:0;left:0;width:100%;height:100%" width="640"></iframe></div>

&nbsp;

&nbsp;

<h2>Glitch example</h2>

&nbsp;

In the server.js, we need to initialise all of our required modules:

&nbsp;

<pre>
<code class="language-javascript">const express = require('express');
const app = express();
const pug = require('pug');

const path = require('path');
const fs = require("fs");
const axios = require('axios');
let FormData = require('form-data');
//formidable takes the form data and saves the file, and parameterises the fields into JSON
const formidable = require('formidable')</code></pre>

&nbsp;

We&#39;ll begin by building the website for our application.&nbsp; We&#39;ll run it using Express with Pug as the page templating&nbsp;tool.&nbsp; The other tools are for handling and creating the forms on the server and sending them to Mindee.

&nbsp;

<h2>Private Token</h2>

&nbsp;

We&#39;ll use environmental variables to store the Mindee API Token:

&nbsp;

<pre>
<code class="language-javascript">let receiptToken = process.env.receiptToken;</code></pre>

&nbsp;

This keeps the token private and secure.&nbsp; In Glitch, if you remix this code, you&#39;ll have to add your token to the .env file for the API call to work correctly.&nbsp;&nbsp;

&nbsp;

<img alt="" src="https://mindee-public-website-staging.s3.amazonaws.com/blog/2020/09/18/screenshot-2020-09-18-at-182558.png" />

&nbsp;

To create your own private token, visit the <a href="https://platform.mindee.net/">Mindee dashboard</a>, create an account, confirm your email, and enter the &quot;Receipt Parsing&quot; section of the dashboard.&nbsp; Click the &quot;Credentials&quot; tab to create your token:

&nbsp;

<img alt="" src="https://mindee-public-website-staging.s3.amazonaws.com/blog/2020/09/03/new_token.jpg" />

&nbsp;

Add this value to the .env file.

&nbsp;

<h2>Webpage</h2>

&nbsp;

The pages we are buiiding here have no CSS, but we&#39;d store that file in the public folder.&nbsp; Just in case you are more ambitious, we&#39;ll open up the public folder available to the browser, and make sure that Pug is the tool used for creating the pages.

&nbsp;

<pre>
<code class="language-javascript">// make all the files in 'public' available
// https://expressjs.com/en/starter/static-files.html
app.use(express.static("public"));
//set up pug as view engine
app.set('view engine','pug');
// https://expressjs.com/en/starter/basic-routing.html
</code></pre>

&nbsp;

&nbsp;The first page is rendered when an initial GET request is made:

&nbsp;

<pre>
<code class="language-javascript">app.get("/", (request, response) =&gt; {
 return response.render('index');
});
</code></pre>

&nbsp;

The index.pug file is a simple page with a form.&nbsp; Users are invited to add a receipt image, and click the upload button. The source is really basic:

&nbsp;

<pre>
<code>head
 link(rel='stylesheet', href='../style.css')
 link(rel='icon', href='icon.ico', type='image/x-icon')
body
 header.header 
 h1 Node.js Receipt parsing

 h2 please upload an image of your receipt

 
 form(action="/" method = "POST" enctype="multipart/form-data").form 
 p
 |Your reciept
 input(type='file', name='imageSource').input 
 input(type='submit', value='Submit')</code></pre>

&nbsp;

This renders the following in your browser:

<img alt="" src="https://mindee-public-website-staging.s3.amazonaws.com/blog/2020/09/18/screenshot-2020-09-18-at-183155.png" />

&nbsp;

When the user adds their receipt and presses submit, the image is sent up to&nbsp;the server:

&nbsp;

<h2>Receipt image management</h2>

&nbsp;

The form sends the image to the server root via POST.&nbsp; When it comes in, formidable reads the form and parses out the files.&nbsp; These files are saved in a temporary directory (with a temporary name). For simplicity, I rename the temp file with the original receipt filename, and send this off to the makeRequest function (that will send the image to Mindee for parsing:

&nbsp;

<pre>
<code class="language-javascript">app.post("/", (request, response) =&gt; {

	//this sends in a form with an image
 //formidable reads the form, saves the image
	let form = new formidable.IncomingForm({maxFileSize : 2000 * 1024 }); //2 MB
	
	form.parse(request, (err, fields, files) =&gt; {
 if (err) {
		 console.error('Error', err);
		 throw err;
 }
	//PARSED FORM
 	console.log("files data", JSON.stringify(files.imageSource));
 let imageName = path.parse(files.imageSource.name).name;
 let imagePath = files.imageSource.path;
 let imageType = files.imageSource.type;
	let imageSize = files.imageSource.size;
	
	//FORMIDIABLE USES A RANDOM NAME ON UPLOAD. RENAME
	let newImagePath = imagePath+ imageName;
 fs.rename(imagePath, newImagePath, function (err) {
 	 if (err) throw err;
 		console.log('File uploaded and moved!');
		//FILE IS RENAMED
		//NOW UPLOAD IT TO MINDEE WITH THE MAKEREQUEST FUNCTION
		makeRequest(newImagePath);
 });
</code></pre>

&nbsp;

&nbsp;

<h2>Receipt extraction</h2>

&nbsp;

The makeRequest function takes the image and sends it to Mindee, receives the response and sends the&nbsp;relevent fields back to the user. The axios library sends the data to Mindee, and we wait for the results:

First we take the image, and place it into a few form called &#39;data; with key &quot;file&quot;.&nbsp;&nbsp;Then we create a request configuration pointing to the Mindee endpoint, insterting our token and the &#39;data&#39; form with the image

&nbsp;

<pre>
<code class="language-javascript">	async function makeRequest(newImagePath) {
 	let data = new FormData()
 		data.append('file', fs.createReadStream(newImagePath))
 		const config = {
 			 method: 'POST',
 			 url: 'https://api.mindee.net/products/expense_receipts/v2/predict',
 			 headers: { 
 				 'X-Inferuser-Token':receiptToken,
 			 ...data.getHeaders()
 				 },
				data
 		}
		console.log("config" ,config);
		try {
			let apiResponse = await axios(config)
			console.log(" api response", apiResponse.data);
		 //pull out the data I want to show on the page
			let predict = apiResponse.data.predictions;
			let merchant = apiResponse.data.predictions[0].merchant.name;
			let merchantType = apiResponse.data.predictions[0].category.value;
			let tax = apiResponse.data.predictions[0].taxes.amount;
			let total = apiResponse.data.predictions[0].total.amount

			console.log (merchant, merchantType, tax, total);
			return response.render('receipt',{merchant, tax, merchantType, total});
				 
		} catch (error) {
 			console.log(error)
 			}

 }
 
 });</code></pre>

&nbsp;

&nbsp;

<h2>Parsing the results</h2>

&nbsp;

The response comes back as a JSON file of data (for full details, <a href="https://mindee.com/documentation/apis/tutorial-parsing-receipts-with-mindees-api">read more here</a>).&nbsp;&nbsp;In&nbsp;this case, we will just extract 4 pieces of data: Merchant, Merchant Type, Taxes and Total.

&nbsp;

I extract these from the API&nbsp;response into&nbsp;variables, and send them&nbsp;to receipt.pug for rendering to the end user.

&nbsp;

<h2>Showing the results</h2>

&nbsp;

The receipt.pug file is again extremely basic, rendering just the four variables on the page, and a link back to the original form.

&nbsp;

<pre>
<code>body
 header.header 
 
 
 
 p here's the data extracted:
	
 p Merchant: #{merchant}
 p Merchant Type: #{merchantType}
 p Tax: #{tax}
 p Total #{total}
 a(href='/') Parse another receipt!</code></pre>

&nbsp;

<img alt="" src="https://mindee-public-website-staging.s3.amazonaws.com/blog/2020/09/20/screenshot-2020-09-20-at-205753.png" />

&nbsp;

If course, you can remix this <a href="https://glitch.com/~grove-tree-sprite">Glitch</a> and change the output to better suit your needs.

&nbsp;

<h2>Conclusion</h2>

In this post, we&#39;ve walked through the steps to create a NodeJS application that receives a receipt, uploads it to Mindee, extracts the results, and presents them back to the user.&nbsp; Give the code a try, and let us know what you think through the chat box at the bottom of the page.

&nbsp;

&nbsp;

You might just want to integrate Mindee's API into your existing NodeJS Service.  In this post, we'll walk through the steps to create a receipt parsing engine running with Mindee's Receipt API and Node.

node.js

Parsing Receipts

<h1>Parsing Invoices with Mindee&rsquo;s API</h1>

&nbsp;

&nbsp;

Scanning and inputting data from invoices into accounting systems is a slow and tedious process.&nbsp; Using Mindee&rsquo;s invoice parsing API, all of the pertinent data can be automatically and accurately extracted from your invoice in seconds, allowing for fast and painless data entry.&nbsp;&nbsp;

&nbsp;

Interested in learning more? In this tutorial, we will walk through the steps to use Mindee&rsquo;s Invoice API.&nbsp;

&nbsp;

<h2>API Prerequisites</h2>

&nbsp;

<ol>
	<li style="list-style-type:decimal">You&rsquo;ll need Mindee account.<a href="http://platform.mindee.net/signup"> Sign up</a><a href="http://platform.mindee.net/signup">&nbsp;for free</a>.&nbsp; Then confirm your email to login.</li>
	<li style="list-style-type:decimal">An invoice.&nbsp; Use a recently received invoice, or do a <a href="https://www.google.com/search?q=invoice">Google Image search</a> for a invoice&nbsp;and download a few to test with.&nbsp;&nbsp;</li>
</ol>

&nbsp;

<h2>Setting up the API</h2>

&nbsp;

Log into you Mindee account and access your Invoices API environment by clicking the Invoices card :

&nbsp;

<h1><img alt="" src="https://mindee-public-website.s3.amazonaws.com/blog/2020/09/11/screenshot-2020-09-11-at-192210.png" style="height:558px; width:400px" /></h1>

To activate the API, click the &ldquo;Try for Free&rdquo; button to access the free tier. You&rsquo;ll land on the dashboard page - this page will show the usage of your account with the Invoice API - of course, it is empty now.&nbsp; On the left navigation, there are links to &ldquo;Documentation&rdquo;, &ldquo;Credentials&rdquo; and &ldquo;Live Interface&rdquo;.&nbsp; The docs tab has all of the technical details you&rsquo;ll need to build for the Invoce API endpoint, and the Live Interface is a cool interactive demo. Rather than try out the demo, we want to build with the API,&nbsp; so click on &ldquo;Credentials&rdquo; to create an API token.

&nbsp;

Add a new token. In this example, I&rsquo;ve named it &ldquo;Tutorial&rdquo;

<h1> 
<img alt="" src="https://mindee-public-website.s3.amazonaws.com/blog/2020/09/01/pasted-image-0-1.png" style="height:247px; width:500px" /></h1>

&nbsp;

Click &ldquo;Add New Key&rdquo; and you&rsquo;ll be able to see your API token.

Now, we are ready to make an API call.&nbsp; In this example, we&rsquo;ll be using cURL.

&nbsp;

<pre>
<code class="language-bash">curl -X POST \ https://api.mindee.net/products/invoices/v1/predict 
 -H 'X-Inferuser-Token: {apiToken}' 
 -H 'content-type: multipart/form-data; boundary=----WebKitFormBoundary7MA4YWxkTrZu0gW' 
 -F file=@/path/to/your/file.png</code></pre>

&nbsp;

&nbsp;

Simply replace ```{apiToken}``` with your new API token and ```/path/to/your/file/png``` with the path to your invoice - wither an image or a pdf.&nbsp;

&nbsp;

NOTE: You can also copy this code right from the documentation tab of the API with your API token inserted for you.

Here&rsquo;s the invoice I used - the roofer who fixed the gasket around our chimney this spring (I&rsquo;ve obfuscated my home address in the image):

&nbsp;

<h1><img alt="" src="https://mindee-public-website.s3.amazonaws.com/blog/2020/09/11/screenshot-2020-09-11-at-193913.png" style="height:383px; width:500px" /></h1>

&nbsp;

Pasting the cURL sample into my terminal, I hit enter and about a second later, I received a JSON response with the receipt details.&nbsp; The full JSON can be accessed here (link to JSON file).&nbsp; Since the response is quite verbose, we will walk through the various fields section by section.

&nbsp;

<h2>Extracted fields</h2>

&nbsp;

<h2>Summary &amp; Documents section</h2>

&nbsp;

The first two sections of the response contain information about the API call made:

<pre>
<code class="language-json">	"call": {
		"endpoint": { "name": "invoices", "version": "1.0" },
		"finished_at": "2020-09-11T18:39:58+00:00",
		"id": "3b4ac75a-f800-466b-bfc4-56c8f8e75134",
		"n_documents": 1,
		"n_inputs": 1,
		"processing_time": 1.047,
		"started_at": "2020-09-11T18:39:57+00:00"
	},
	"documents": [
		{
			"id": "5d72b0c2-16dd-481d-9e8b-031c5d1ab33a",
			"name": "Screenshot 2020-09-11 at 19.39.13.png"
		}
	],</code></pre>

&nbsp;

The call section tells us that we ran on the v1 of the invoice endpoint, uploading one document that is one page long.&nbsp; After a second, the file was processed, and the response transmitted back to me.&nbsp; The documents section gives the Mindee id for the file, and the filename (and you can tell I used a screenshot of the original invoice).

&nbsp;

<h2>Predictions:&nbsp;</h2>

&nbsp;

This is the fun part, where the API extraxcts pertinent details form teh invoice automatcilly, saving tedious manual data entry.&nbsp; Every invoice is different, and will have different fields. This invoice, being sent from the UNited States will not have some common European features like and IBAN or Tax Id.&nbsp;&nbsp;

Other peices are rpesent in this invoice, but not picked up by the API.&nbsp; We are planning multiple additional releases to this API in 2020 that will continue to improve the accuracy and precision of the results.&nbsp;&nbsp;

&nbsp;

<h3>Company Number</h3>

In this case, there is no company number in the invoice:

<pre>
<code class="language-json">"company_number": {
				"probability": 0,
				"segmentation": { "bounding_box": [] },
				"type": "N/A",
				"value": "N/A"
			},</code></pre>

<h2>&nbsp;</h2>

<h3>Due Date</h3>

My invoice does have a due date, but it was not extracted.&nbsp;

<pre>
<code class="language-json">			"due_date": {
				"iso": "N/A",
				"probability": 0,
				"raw": "N/A",
				"segmentation": { "bounding_box": [] }
			},</code></pre>

<h2>&nbsp;</h2>

<h3>Invoice Date</h3>

The work was done in late June of this year, and the API correctly extracts the invoice date, along with four [x,y] points boxing in the value:

<pre>
<code class="language-json">			"invoice_date": {
				"iso": "2020-06-29",
				"probability": 0.99,
				"segmentation": {
					"bounding_box": [
						[ 0.766, 0.392 ],
						[ 0.855, 0.392 ],
						[ 0.855, 0.416 ],
						[ 0.766, 0.416 ]
					]
				}
			},</code></pre>

<h3>&nbsp;</h3>

<h3>Invoice Number</h3>

The invoice is numbered 1277, which is extracted by the API, along with the four [x,y] corrdinates denoting the location in the image.

<pre>
<code class="language-json">			"invoice_number": {
				"probability": 0.99,
				"segmentation": {
					"bounding_box": [
						[ 0.766, 0.363 ],
						[ 0.807, 0.363 ],
						[ 0.807, 0.382 ],
						[ 0.766, 0.382 ]
					]
				},
				"value": "1277"
			},</code></pre>

<h3>&nbsp;</h3>

<h3>Locale</h3>

Based on the information on the invoice, the API is able to predict (with 82% confidence) that this bill originated in the United States, is in USD, and is in English.

<pre>
<code class="language-json">"locale": { "currency": "USD", "language": "en", "probability": 0.82 },</code></pre>

<h2>&nbsp;</h2>

<h3>Orientation</h3>

The invoice did not require rotation to be parsed.

<pre>
<code class="language-json">"orientation": { "degrees": 0, "probability": 0.99 },</code></pre>

<h2>&nbsp;</h2>

<h3>Payment Details</h3>

The invoice does not contain any payment details, so were not extracted.

<pre>
<code class="language-json">"payment_details": {
				"iban": "N/A",
				"probability": 0,
				"segmentation": { "bounding_box": [] }
			},</code></pre>

<h3>&nbsp;</h3>

<h3>Supplier</h3>

The API is only 50% confident in this result, but it does extract the name of the company (despite without any spaces in the name).&nbsp;&nbsp;

<pre>
<code class="language-json">			"supplier": {
				"probability": 0.5,
				"segmentation": {
					"bounding_box": [
						[ 0.446, 0.066 ],
						[ 0.561, 0.066 ],
						[ 0.561, 0.083 ],
						[ 0.446, 0.083 ]
					]
				},
				"value": "SILVERHAMMER"
			},</code></pre>

<h3>&nbsp;</h3>

<h3>Tax Id</h3>

Companies in the US generally do not list their Tax IDs on invoices, so this was not found.

<pre>
<code class="language-json">			"tax_id": {
				"probability": 0,
				"segmentation": { "bounding_box": [] },
				"value": "N/A"
			},</code></pre>

<h3>&nbsp;</h3>

<h3>Taxes</h3>

The API did not extract the texes, or the percentage tax rate (we have an update later this year t better extract US tax details.)

<pre>
<code class="language-json">"taxes": [],</code></pre>

<h2>&nbsp;</h2>

<h3>Total without Taxes</h3>

The API did not extract the taxes, but did get the cost before taxes with 99% confidence.&nbsp;&nbsp;

<pre>
<code class="language-json">	"total_excl": {
				"amount": 350,
				"probability": 0.99,
				"segmentation": {
					"bounding_box": [
						[ 0.874, 0.803 ],
						[ 0.931, 0.803 ],
						[ 0.931, 0.825 ],
						[ 0.874, 0.825 ]
					]
				}
			},</code></pre>

<h2>&nbsp;</h2>

<h3>Total with Taxes</h3>

The total with taxes was alos extracted accurately with 99% confidence, meaning that the tax value could easily be calculated with a simple difference.

<pre>
<code class="language-json">"total_incl": {
				"amount": 380.45,
				"probability": 0.99,
				"segmentation": {
					"bounding_box": [
						[ 0.842, 0.897 ],
						[ 0.933, 0.897 ],
						[ 0.933, 0.93 ],
						[ 0.842, 0.93 ]
					]
				}
			}</code></pre>

&nbsp;

<h2>Conclusion</h2>

Manual entry of invoices is a time consuming task. Adding the Mindee API to your system will simplify the extraction and make your accounts paybale processes run more accurately and smoothly.&nbsp; Do you have more questions? Click the chat button at the bottom right.

&nbsp;

<h1>&nbsp;</h1>

&nbsp;

&nbsp;

Inputting data from invoices into accounting systems is a slow and tedious process.  Using Mindee’s invoice parsing API, extract data from invoices automatically and accurately in seconds, allowing for fast and painless data entry.

<h1>Extract invoice&nbsp;details&nbsp;with Mindee&rsquo;s Invoice API and NodeJS</h1>

&nbsp;

In&nbsp;this example, we will build from scratch a NodeJS based web application that accepts an&nbsp;image or PDF of an invoice, and return&nbsp;several points of data extracted from the invoice.

&nbsp;

The code for the sample code is availble on <a href="https://github.com/dougsillars/mindee_parse_invoice">Github</a>, and also available on <a href="https://glitch.com/edit/#!/recondite-icy-wanderer?path=server.js%3A13%3A4">Glitch</a>.&nbsp; For simplicity, we&#39;ll follow along in the Glitch example.

&nbsp;

<div class="youtube-embed-wrapper" style="height:0; overflow:hidden; padding-bottom:56.25%; padding-top:30px; position:relative"><iframe frameborder="0" height="360" src="https://www.youtube.com/embed/0lV_PtELbqk" style="position:absolute;top:0;left:0;width:100%;height:100%" width="640"></iframe></div>

&nbsp;

<h2>Glitch example</h2>

&nbsp;

In the server.js, we need to initialise all of our required modules:

&nbsp;

<pre>
<code class="language-javascript">const express = require('express');
const app = express();
const pug = require('pug');

const path = require('path');
const fs = require("fs");
const axios = require('axios');
let FormData = require('form-data');
//formidable takes the form data and saves the file, and parameterises the fields into JSON
const formidable = require('formidable')
</code></pre>

&nbsp;

We&#39;ll begin by building the website for our application.&nbsp; We&#39;ll run it using Express with Pug as the page templating&nbsp;tool.&nbsp; The other tools are for handling and creating the forms on the server and sending them to Mindee.

&nbsp;

<h2>Private Token</h2>

&nbsp;

We&#39;ll use environmental variables to store the Mindee API Token:

&nbsp;

<pre>
<code class="language-javascript">let invoiceToken = process.env.invoiceToken;</code></pre>

&nbsp;

This keeps the token private and secure.&nbsp; In Glitch, if you remix this code, you&#39;ll have to add your token to the .env file for the API call to work correctly.&nbsp;&nbsp;

&nbsp;

<img alt="" src="https://mindee-public-website-staging.s3.amazonaws.com/blog/2020/09/24/screenshot-2020-09-24-at-121954.png" style="height:158px; width:400px" />

&nbsp;

To create your own private token, visit the <a href="https://platform.mindee.net/">Mindee dashboard</a>, create an account, confirm your email, and enter the &quot;Invoice&nbsp;API&quot; section of the dashboard.&nbsp; Click the &quot;Credentials&quot; tab to create your token:

&nbsp;

<img alt="" src="https://mindee-public-website-staging.s3.amazonaws.com/blog/2020/09/03/new_token.jpg" />

&nbsp;

Add this value to the .env file.

&nbsp;

<h2>Webpage</h2>

&nbsp;

The pages we are buiiding here have no CSS, but we&#39;d store that file in the public folder.&nbsp; Just in case you are more ambitious, we&#39;ll open up the public folder available to the browser, and make sure that Pug is the tool used for creating the pages.

&nbsp;

<pre>
<code class="language-javascript">// make all the files in 'public' available
// https://expressjs.com/en/starter/static-files.html
app.use(express.static("public"));
//set up pug as view engine
app.set('view engine','pug');
// https://expressjs.com/en/starter/basic-routing.html
</code></pre>

&nbsp;

&nbsp;The first page is rendered when an initial GET request is made:

&nbsp;

<pre>
<code class="language-javascript">app.get("/", (request, response) =&gt; {
 return response.render('index');
});
</code></pre>

&nbsp;

The index.pug file is a simple page with a form, where users can add an invoice&nbsp;image, and click the upload button. Not a lot to see here:

&nbsp;

<pre>
<code class="language-html">head
 link(rel='stylesheet', href='../style.css')
 link(rel='icon', href='icon.ico', type='image/x-icon')
body
 header.header 
 
 
 h1 Node.js Invoice extraction

 h2 please upload an image or PDF of your invoice

 
 form(action="/" method = "POST" enctype="multipart/form-data").form 
 p
 |Your reciept
 input(type='file', name='imageSource').input 
 input(type='submit', value='Submit')</code></pre>

&nbsp;

This renders the following in your browser:

&nbsp;

<img alt="" src="https://mindee-public-website-staging.s3.amazonaws.com/blog/2020/09/24/screenshot-2020-09-24-at-122314.png" style="height:290px; width:300px" />

&nbsp;

When the user adds an invoice and presses submit, the image is sent up to the server for processesing

&nbsp;

<h2>Invoice image processing on the server</h2>

&nbsp;

The form sends the image to the server root via POST.&nbsp; When it comes in, formidable reads the form and parses out the files.&nbsp; These files are saved in a temporary directory (with a temporary name). For simplicity, I rename the temp file with the original invoice filename.&nbsp; I do this as the filename in the API response will match whatever is uploaded to Mindee, so keeping the filename the same makes identification&nbsp;easier.&nbsp;The file is then sent to the&nbsp;makeRequest function:

&nbsp;

<pre>
<code class="language-javascript">app.post("/", (request, response) =&gt; {

	//this sends in a form with an image
 //formidable reads the form, saves the image
	let form = new formidable.IncomingForm({maxFileSize : 2000 * 1024 }); //2 MB
	
	form.parse(request, (err, fields, files) =&gt; {
 if (err) {
		 console.error('Error', err);
		 throw err;
 }
	//PARSED FORM
 	console.log("files data", JSON.stringify(files.imageSource));
 let imageName = path.parse(files.imageSource.name).name;
 let imagePath = files.imageSource.path;
 let imageType = files.imageSource.type;
	let imageSize = files.imageSource.size;
	
	//FORMIDIABLE USES A RANDOM NAME ON UPLOAD. RENAME
	let newImagePath = imagePath+ imageName;
 fs.rename(imagePath, newImagePath, function (err) {
 	 	if (err) throw err;
 		 console.log('File uploaded and moved!');
		 //FILE IS RENAMED
		 //NOW UPLOAD IT TO MINDEE WITH THE MAKEREQUEST FUNCTION
		 makeRequest(newImagePath);
 	});
	</code></pre>

&nbsp;

<h2>Invoice data extraction</h2>

&nbsp;

The makeRequest function takes the image and sends it to Mindee, receives the response and sends the&nbsp;relevent fields back to the user. The axios library sends the data to Mindee, and we wait for the results:

First we take the image, and place it into a few form called &#39;data; with key &quot;file&quot;.&nbsp;&nbsp;Then we create a request configuration pointing to the Mindee endpoint, insterting our token and the &#39;data&#39; form with the image

&nbsp;

<pre>
<code class="language-javascript">		async function makeRequest(newImagePath) {
 		 let data = new FormData()
 		 data.append('file', fs.createReadStream(newImagePath))
	
		//	console.log("form data ", data);
	
 		 const config = {
 			 method: 'POST',
 			 url: 'https://api.mindee.net/products/invoices/v1/predict',
 			 headers: { 
 				 'X-Inferuser-Token':invoiceToken,
 				 ...data.getHeaders()
 				 },
				 data
 			 }
			 console.log("config" ,config);
			 try {
				 let apiResponse = await axios(config)
				 console.log(" api response", apiResponse.data);
				 
			</code></pre>

&nbsp;

&nbsp;

<h2>Results</h2>

&nbsp;

The response comes back as a JSON file of data that has been extracted from the invoice (for full details, [read more here](https://mindee.com/documentation/apis/invoice-parsing)).&nbsp;&nbsp;We&#39;ll extract the invoice number, date, company name, the currency, and the total before and after taxes.

&nbsp;

I extract these from the APII response into&nbsp;variables, and send them&nbsp;to invoice.pug for rendering to the end user.

&nbsp;

<pre>
<code class="language-javascript"> //pull out the data I want to show on the page
let currency = apiResponse.data.predictions[0].locale.currency;
let invoice = apiResponse.data.predictions[0].invoice_number.value;
let merchant = apiResponse.data.predictions[0].supplier.value;
let date = apiResponse.data.predictions[0].invoice_date.iso;
let beforeTax = apiResponse.data.predictions[0].total_excl.amount;
let total = apiResponse.data.predictions[0].total_incl.amount
				 
console.log (invoice, currency, merchant, date, beforeTax, total);
return response.render('invoice',{invoice, currency, merchant, date, beforeTax, total});
				</code></pre>

&nbsp;

&nbsp;

<h2>Displaying the data</h2>

&nbsp;

The invoice.pug file renders&nbsp;just the six&nbsp;results we extracted&nbsp;and a link back to the original form.

&nbsp;

<pre>
<code>body
 header.header 
 
 
 
 p here's the data extracted:
	
 p Merchant: #{merchant}
 p invoice number: #{invoice}
 p Date: #{date}
 p Currency: #{currency}
 p Total before Taxes: #{beforeTax}
 p Total with tax #{total}
 a(href='/') Parse another invoice!</code></pre>

&nbsp;

<img alt="" src="https://mindee-public-website-staging.s3.amazonaws.com/blog/2020/09/24/screenshot-2020-09-24-at-124750.png" style="height:327px; width:300px" />

&nbsp;

&nbsp;

If course, you can remix this <a href="https://glitch.com/~recondite-icy-wanderer">Glitch</a> and change the output to better suit your needs.

&nbsp;

&nbsp;

<h2>Conclusion</h2>

In this post, we&#39;ve walked through the steps to create a NodeJS application that receives an invoice, uploads it to Mindee, extracts the results, and presents them back to the user.&nbsp; Give the code a try, and let us know what you think through the chat box at the bottom of the page.

&nbsp;

&nbsp;

The Invoice parsing API from Mindee is a powerful tool to quickly extract crucial data from invoices. In this post, we walk though a basic NodeJS integration.

Parsing Invoices

<h1>Parsing international passports with Mindee&#39;s API</h1>

&nbsp;

Many onboarding processes in mobile or web apps require to extract some data from ID documents. In this tutorial, you will learn how to automatically extract data from passports to offer to your users the best onboarding experience.

&nbsp;

We will walk through the steps to use Mindee&rsquo;s Internation Passport&nbsp;Parsing API.&nbsp; Let&rsquo;s get started!&nbsp;

&nbsp;

<h2>API Prerequisites</h2>

<ol>
	<li>You&rsquo;ll need a free Mindee account. <a href="https://platform.mindee.net/signup" target="_blank">Sign up</a>&nbsp;and confirm your email to login.</li>
	<li>A picture&nbsp;of a passport.&nbsp; You can use yours&nbsp;safely as data protection is one of our priority and we won&#39;t send your images to any third party application. You can also download a fake one <a href="https://mindee-public-website.s3.amazonaws.com/blog/2020/09/03/fake_passport.jpg">here</a>.</li>
</ol>

&nbsp;

&nbsp;

<img alt="" src="https://mindee-public-website-dev.s3.amazonaws.com/blog/2020/08/07/passport_pic.jpg" />

&nbsp;

&nbsp;

<h2>Setting up the API</h2>

&nbsp;

Log into your Mindee account and access your passport API environment by clicking the International&nbsp;Passport&nbsp;card:

&nbsp;

&nbsp;

<img alt="passport api mindee" src="https://mindee-public-website-dev.s3.amazonaws.com/blog/2020/08/07/passport_article.jpg" />

&nbsp;

To activate the API, click the &ldquo;Try for Free&rdquo; button. This will give you access the API for free 50 times a month.&nbsp; There are four sections on the API landing page, noted in the left navigation. You are currently on the dashboard, and there&nbsp;are additional links to &ldquo;Documentation&rdquo;, &ldquo;Credentials&rdquo; and &ldquo;Live Interface&rdquo;.&nbsp; The docs tab has all of the technical details you&rsquo;ll need to build for the passports API endpoint, and the Live Interface is a cool interactive demo.

We&#39;ll start by clicking &quot;credentials&quot; and creating a new API token, in this case, named Tutorial:

&nbsp;

<img src="https://mindee-public-website.s3.amazonaws.com/blog/2020/09/01/pasted-image-0-1.png" style="height:198px; width:400px" />

&nbsp;

Click &ldquo;Add New Key&rdquo; and you&rsquo;ll be able to see your API token,.

&nbsp;

If you move back to the documentation tab, you can pick your API token, and the language, and create an API call.&nbsp; Here is the API call in cURL:

<pre>
<code class="language-bash">curl -X POST \ https://api.mindee.net/products/passport/v1/predict \ 
-H 'X-Inferuser-Token: {apiToken}’ \
-H 'content-type: multipart/form-data; boundary=----WebKitFormBoundary7MA4YWxkTrZu0gW' \
-F file=@/path/to/your/file.png</code></pre>

&nbsp;

Simply replace {apiToken}&nbsp;with your new API token and /path/to/your/file/png with the path to your receipt.&nbsp;

 
That&#39;s it! We&#39;re all set to run this command in the terminal, and see the data that Mindee can extract.

In this tutirla, I&#39;ll use a sample [Cyprus passport](https://commons.wikimedia.org/wiki/File:Cyprus_passport_data_page.jpg) from Wikimedia.

<img alt="Cyprus Passport used in this example" src="https://mindee-public-website.s3.amazonaws.com/blog/2020/09/04/1024px-cyprus_passport_data_page.jpg" style="height:334px; width:514px" />

<h2>&nbsp;</h2>

<h2>Extracted fields</h2>

&nbsp;

The JSON response can be a bit verbose, so we will break it into sections to describe what we are seeing.&nbsp; The initial sections describe the API and the API response:

<pre>
<code class="language-json">"call": {
		"endpoint": {
			"name": "passport",
			"version": "1.0"
		},
		"finished_at": "2020-09-04T20:57:51+00:00",
		"id": "7d81f5fe-fce1-4b63-b3de-cc5948c39b55",
		"n_documents": 1,
		"n_inputs": 1,
		"processing_time": 1.988,
		"started_at": "2020-09-04T20:57:49+00:00"
	},
	"documents": [
		{
			"id": "57aded44-a216-45be-859c-f949cc5b011d",
			"name": "1024px-Cyprus_passport_data_page.jpg"
		}
	],</code></pre>

Here, we can see that passport API was called, and it processed one document in about 2 seconds.&nbsp; What we are really interested in seeing is the results, and these can be found in the predictions section of the JSON file:

<h2>&nbsp;</h2>

<h2>Predictions</h2>

&nbsp;

The API response is in alphabetical order of items extracted from the document:

&nbsp;

<h3>Birth Date</h3>

<pre>
<code class="language-json">"birth_date": {
				"probability": 1.0,
				"segmentation": {
					"bounding_box": [
						[.053,	0.912],
						[0.955,0.912],
						[0.955,0.969],
						[0.053,0.969]
					]
				},
				"value": "1970-01-01"
			},</code></pre>

The API matched the brithdate in the MRX fields (more on that in a minute), so is 100% confident that the birthdate is 1 January, 1970. The bounding box gives four (x,y) points in the image where the date of birth can be found.

&nbsp;

<h3>Country of issuance</h3>

The API then attempts to parse the country that issued the passport.&nbsp;

<pre>
<code class="language-json">"country": {
				"probability": 0.99,
				"segmentation": {
					"bounding_box": [
						[0.479,0.199],
						[0.522,0.199],
						[0.522,0.234],
						[0.479,0.234]
					]
				},
				"value": "CYP"
			},</code></pre>

&nbsp;

The API is 99% certain that Cyprus is the issuing country. This is really incredible, as the API team tells me that there are no Cyprus passports in the data training set. Again, 4 (x,y) coordinates in the image point out where CYP is used.

<h3>&nbsp;</h3>

<h3>Expiration Date</h3>

<pre>
<code class="language-json">"expiry_date": {
				"probability": 1.0,
				"segmentation": {
					"bounding_box": [
						[0.053,0.912],
						[0.955,0.912],
						[0.955,0.969],
						[0.053,0.969]
					]
				},
				"value": "2020-12-01"
			},</code></pre>

Many countries do not allow entry with under six months of validity left on the passport.&nbsp; This woman should probably start the renewal process,&nbsp; as the API has identified that this passport will expire on 1 December, 2020.

<h3>&nbsp;</h3>

<h3>Gender</h3>

<pre>
<code class="language-json">			"gender": {
				"probability": 0.1,
				"segmentation": {
					"bounding_box": [
						[0.053,0.912],
						[0.955,0.912],
						[0.955,0.969],
						[0.053,0.969]
					]
				},
				"value": "F"
			},</code></pre>

The API&#39;s confidence is not high on this one, just 10%, but it does correctly identify the gender as female.

<h3>&nbsp;</h3>

<h3>Given Name</h3>

<pre>
<code class="language-json">			"given_names": [
				{
					"probability": 0.99,
					"segmentation": {
						"bounding_box": [
							[0.048,0.844],
							[0.952,0.844],
							[0.952,0.895],
							[0.048,0.895]
						]
					},
					"value": "AFRODITI"
				}
			],</code></pre>

The API is really certain (99% confidence) that the given name on the passport is Afroditi, meaning that our passport holder was named after the Greek goddess of beauty and love.

<h3>&nbsp;</h3>

<h3>ID Number</h3>

<pre>
<code class="language-json">			"id_number": {
				"probability": 1.0,
				"segmentation": {
					"bounding_box": [
						[0.673,0.197],
						[0.774,0.197],
						[0.774,0.233],
						[0.673,0.233]
					]
				},
				"value": "K00000413"
			},</code></pre>

While we are not numbers, in many ways, we are identified by numbers. In this case, the passport ID number, which must be used when booking flights, checking into hotels, etc.&nbsp; The API has found the location (marked by the 4 (x,y) points in the bounding box, for the Passport number, which is K00000413.

&nbsp;

<h3>Issuance Date</h3>

Contining the alphabetical trip through the passport fields, we next come to the date the passport was issued:

&nbsp;

<pre>
<code class="language-json">			"issuance_date": {
				"probability": 0.18,
				"segmentation": {
					"bounding_box": [
						[0.353,0.586],
						[0.426,0.586],
						[0.426,0.62],
						[0.353,0.62]
					]
				},
				"value": "2014-01-21"
			},</code></pre>

In this case, the API missed the issuance date.&nbsp; However, while our API does miss in it&#39;s predictions, in this case, the issue is due to the fact that the passport is a sample (we&#39;ll have a blog post on this soon).&nbsp; The issuance date is missing from the MRZ zone of the sample&nbsp;passport, so the API is unable to accurately measure the issuance date.

&nbsp;

<h3>Surname</h3>

<pre>
<code class="language-json">"surname": {
				"probability": 0.98,
				"segmentation": {
					"bounding_box": [
						[0.33,0.272],
						[0.444,0.272],
						[0.444,0.309],
						[0.33,0.309]
					]
				},
				"value": "ANONYMOU"
			}</code></pre>

Our fake passport holder has a surname of Anonymou, which is correctly identified by the API, along with bounding boxes.

&nbsp;

<h3>MRZ1 and MRZ2</h3>

The last two regions identified by the API (which I pulled *slightly* out of alphabetical order for readability) are the 2 Machine Readable Zone (MRZ) lines:

<pre>
<code class="language-json">"mrz1": {
				"probability": 0.99,
				"segmentation": {
					"bounding_box": [
						[0.048,0.844],
						[0.952,0.844],
						[0.952,0.895],
						[0.048,0.895]
					]
				},
				"value": "P&lt;CYPANONYMOU&lt;&lt;AFRODITI&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;"
			},
"mrz2": {
				"probability": 0.1,
				"segmentation": {
					"bounding_box": [
						[0.053,0.912],
						[0.955,0.912],
						[0.955,0.969],
						[0.053,0.969]
					]
				},
				"value": "K000004134CYP7001017F2012010&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;4"
			},</code></pre>

The MRZ zones (if you cannot tell from the JSON above) are the two lines of machine readable text at the bottom of every passport page.&nbsp; MRZ1 is line one, and MRZ2 is line 2.&nbsp; Most of the identifying information can be pulled from this area.&nbsp; The confidence interval for line one is high, as the fake passport has this area filled out correctly.&nbsp; However, the confidence level is low, as the algorithm expected more characters other than &quot;&lt;&quot; to supply the issuance date.

&nbsp;

<h2>Conclusion</h2>

&nbsp;

And there you have it! Mindee&#39;s passport API parsed a passport in under 2 seconds, matching nearly all of the sections of the passport quickly and accurately. Give it&nbsp;try with our free tier, and let us know in the chat how it works for you!

Many onboarding processes in mobile or web apps require to extract some data from ID documents. In this tutorial, you will learn how to automatically extract data from passports to offer to your users the best onboarding experience.

<h1>Extract data from passports with Mindee&#39;s API using Python&nbsp;</h1>

&nbsp;

In this tutorial, you will learn how to&nbsp;extract data from passports (pdfs or images)&nbsp;using python. This can help you automatically process id document for user onboarding or KYC processes.

&nbsp;

If you want to get more details about how to use the mindee python library for passports, please see:

<ul>
	<li>1.&nbsp;<a href="https://mindee.com/documentation/apis/tutorial-parsing-passports-with-mindees-api/passport-python/getting-started">Getting started with passport parsing in python</a></li>
	<li>2.&nbsp;<a href="https://mindee.com/documentation/apis/tutorial-parsing-passports-with-mindees-api/passport-python/passport-fields">Extracted passport fields</a></li>
	<li>3.&nbsp;<a href="https://mindee.com/documentation/apis/tutorial-parsing-passports-with-mindees-api/passport-python/save-and-restore">Save and restore passport API responses</a></li>
	<li>4.&nbsp;<a href="https://mindee.com/documentation/apis/tutorial-parsing-passports-with-mindees-api/passport-python/passport-data-validation">Passport data validations</a></li>
	<li>5.&nbsp;<a href="https://mindee.com/documentation/apis/tutorial-parsing-passports-with-mindees-api/passport-python/performances-benchmark">Benchmark Passport OCR performances</a></li>
</ul>

&nbsp;

&nbsp;

<h2>Tutorial</h2>

&nbsp;

In this tutorial, you will learn how to extract passport data automatically using Mindee python helper library,&nbsp;transform a pdf into an image file, and how to&nbsp;highlight features&nbsp;on it to make human validation easier.&nbsp;

&nbsp;

The final result looks like this:

&nbsp;

<img alt="" src="https://mindee-public-website-dev.s3.amazonaws.com/blog/2020/11/04/ezgifcom-video-to-gif-17.gif" />

&nbsp;

&nbsp;

<h2>API Prerequisites</h2>

&nbsp;

<ol>
	<li>You&rsquo;ll need a free Mindee account.&nbsp;<a href="https://platform.mindee.net/signup">Sign up</a>&nbsp;and confirm your email to login.</li>
	<li>A passport image or pdf. You can&nbsp;download one&nbsp;<a href="https://mindee-public-website-dev.s3.amazonaws.com/blog/2020/11/04/fake_passport-1.jpg" target="_blank">here</a>&nbsp;or&nbsp;<a href="https://www.google.com/search?q=fake+passport">check out on google image</a></li>
</ol>

&nbsp;

&nbsp;

<h2>Setup the project</h2>

&nbsp;

Create an empty directory wherever you want in your laptop, we&rsquo;ll call ours &ldquo;python_passport&rsquo;, and create a &lsquo;main.py&rsquo; file in it.

&nbsp; 
We recommend you to create a virtual environment before going to the next step, in order to avoid installing the required packages globally on your machine.To do so, run the following command:

&nbsp;

(macOS and Linux)

<pre>
<code class="language-bash">python -m venv venv</code></pre>

 
(windows)

<pre>
<code class="language-bash">py -m venv venv</code></pre>

&nbsp;

Then, you&#39;ll need to activate your venv, so that pip will install packages in that folder.

&nbsp;

(macOS and Linux)

<pre>
<code class="language-bash">source venv/bin/activate</code></pre>

 
(Windows)

<pre>
<code class="language-bash">.\env\Scripts\activate</code></pre>

&nbsp;

&nbsp;

We need now to Install mindee python library&nbsp;from PyPi using pip, a package manager for Python.

&nbsp;

<pre>
<code class="language-bash">pip install mindee</code></pre>

&nbsp;

If the install throws an error, you might need to upgrade your version of pip. To do so, run:

&nbsp;

<pre>
<code class="language-bash">pip install --upgrade pip </code></pre>

&nbsp;

Don&#39;t have pip installed? Try installing it, by running this from the command line:

&nbsp;

<pre>
<code class="language-bash">$ curl https://bootstrap.pypa.io/get-pip.py | python</code></pre>

&nbsp;

That&rsquo;s it for the setup, let&rsquo;s call the API.

&nbsp;

&nbsp;

<h2>Call the Mindee Passport API</h2>

&nbsp;

&nbsp;

Login onto the <a href="https://platform.mindee.net/">Mindee dashboard</a>&nbsp;and enter the&nbsp;Passport API&nbsp;environment by clicking the following card.

&nbsp;

&nbsp;

<img alt="mindee invoice api card" src="https://mindee-public-website-dev.s3.amazonaws.com/blog/2020/11/04/passport_article.jpg" />

&nbsp;

&nbsp;

If you don&rsquo;t have an API token yet for this API, go to the &ldquo;credentials&rdquo; section and create a new API Key.&nbsp;

&nbsp;

&nbsp;

<img alt="new api token" src="https://lh3.googleusercontent.com/CWRZZFfY2uVd5Lyx-wxxR_EjWcw6ZBryFhhpNMwP1DBO5LQEEUBfqos2sPjCE4BQ6AdxelSzaybQEl5EGGXDL-eqwr5qXA3dtgn8_TquDVfVp8xFMsjV_npDbTblZ-wXTgOkwoEy" />

&nbsp;

&nbsp;

Open the &ldquo;main.py&rdquo; file you created a few minutes ago, and paste the following&nbsp;code:

&nbsp;

&nbsp;

<pre>
<code>from mindee import Client

filepath = "./path/to/passport.jpg"
passport_token = "my-token-here"

mindee_client = Client(passport_token=passport_token)

if __name__ == "__main__":

 response = mindee_client.parse_passport(filepath)

 print(response.passport)</code></pre>

&nbsp;

&nbsp;

Replace the &ldquo;path/to/my/file&rdquo; placeholder in the code with the path of the passport pdf or image you want to parse. Replace the &ldquo;my-token-here&rdquo; with the API token you created previously in the platform.

&nbsp;

&nbsp;

You can get back to your console and run the script

&nbsp;

&nbsp;

<pre>
<code>python main.py</code></pre>

&nbsp;

&nbsp;

You should see the details of the passport printed in your console:

&nbsp;

<pre>
<code class="language-apache">-----Passport data-----
Filename: passport.jpg 
Full name: HENERT PUDARSAN 
Given names: HENERT 
Surname: PUDARSAN
Country: GBR
ID Number: 707797979
Issuance date: 2012-04-22
Birth date: 1995-05-20
Expiry date: 2017-04-22
MRZ 1: P&lt;GBRPUDARSAN&lt;&lt;HENERT&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;
MRZ 2: 7077979792GBR9505209M1704224&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;00
MRZ: P&lt;GBRPUDARSAN&lt;&lt;HENERT&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;7077979792GBR9505209M1704224&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;00
----------------------</code></pre>

&nbsp;

Now let&rsquo;s take a look at the passport object.

&nbsp;

<h2>&nbsp;</h2>

<h2>Extracted passport data</h2>

&nbsp;

The API returns a list of different fields in the passport (given names, surname, date of birth &hellip;). You can find this list in the <a href="https://mindee.com/documentation/apis/tutorial-parsing-passports-with-mindees-api/passport-python">Extracted passport fields</a> documentation

&nbsp;

The passport object under the&nbsp;<code>Response.passport</code>&nbsp;attribute has all the features you need to get the passport data and coordinates in the image to highlight the results.

&nbsp;

Here is a piece of code to show you how to get some of them:

&nbsp;

<pre>
<code class="language-python">from mindee import Client

filepath = "./path/to/passport.jpg"
passport_token = "my-token-here"

mindee_client = Client(passport_token=passport_token)

if __name__ == "__main__":
 response = mindee_client.parse_passport(filepath)

 print(response.passport)

 # To get the list of names
 print(response.passport.given_names)

 # Loop on each given name
 for given_name in response.passport.given_names:
 # To get the name string
 print(given_name.value)

 # To get the passport's owner surname (string)
 print(response.passport.surname.value)

 # To get the passport's owner gender (string among {"M", "F"}
 print(response.passport.gender.value)

 # To get the passport's owner full name (string)
 print(response.passport.full_name.value)

 # To get the passport's owner birth place (string)
 print(response.passport.birth_place.value)

 # To get the passport's owner date of birth (string)
 print(response.passport.birth_date.value)

 # To get the passport expiry date (string)
 print(response.passport.expiry_date.value)

 # To get the passport date of issuance (string)
 print(response.passport.issuance_date.value)

 # To get the passport first line of machine readable zone (string)
 print(response.passport.mrz1.value)

 # To get the passport second line of machine readable zone (string)
 print(response.passport.mrz2.value)

 # To get the passport full machine readable zone (string)
 print(response.passport.mrz.value)

 # To get the passport id number (string)
 print(response.passport.id_number.value)

 # To get the passport country code (string)
 print(response.passport.country_code.value)</code></pre>

&nbsp;

Run the script and check out the results in your console!

&nbsp;&nbsp;

Now we are going to add a piece of code that will highlight the features extracted on the pdf. It can help you or your users to very quickly validate the data extraction.

&nbsp;

&nbsp;

<h2>Convert pdf into an image</h2>

&nbsp;

If you&rsquo;re using a pdf,&nbsp;we need first to convert the pdf into an image, as it&rsquo;s more convenient to use for highlighting the features on it. If your passport is an image, you can skip this part.

&nbsp;

To convert the pdf into a jpeg we are going to use the&nbsp;<a href="https://github.com/Belval/pdf2image">pdf2image</a>&nbsp;python library; First install it in your environment:

&nbsp;

<pre>
<code>pip install pdf2image</code></pre>

&nbsp;

Important note: Dealing with pdfs is not easy and this library might not work directly because it sometimes requires some extra dependencies. Check out the github to make sure you installed it correctly.

&nbsp;

Once installed, we are going to create an image out of our pdf:

&nbsp;

&nbsp;

<pre>
<code>from pdf2image import convert_from_path, convert_from_bytes
images = convert_from_path(filepath)</code></pre>

&nbsp;

&nbsp;

The images object is an array of Pillow images. Pillow is an image library for python. As we want to use openCV for highlighting the features on the image, we need to convert the pillow image into an OpenCV image. OpenCV is based on the numpy library, and to convert it we are going to use numpy directly.

&nbsp;

If you don&rsquo;t have numpy installed on your environment:

&nbsp;

&nbsp;

<pre>
<code>pip install numpy</code></pre>

&nbsp;

&nbsp;

Now let&rsquo;s convert the Pillow image into an opencv format using numpy:

&nbsp;

&nbsp;

<pre>
<code>import numpy as np
from pdf2image import convert_from_path
 
def pdf_to_cv_image(filepath, page=0):
 image = convert_from_path(filepath)[page]
 return np.asarray(image)</code></pre>

&nbsp;

 
 
&nbsp;

<h2>Highlight features on the image</h2>

&nbsp;

Let&rsquo;s try to highlight the features as if someone did it with a pen.

&nbsp;

First, you&rsquo;ll need to install the computer vision python library OpenCV if you don&rsquo;t have it already installed in your env. To do so, run:

&nbsp;

&nbsp;

<pre>
<code>pip install opencv-python</code></pre>

&nbsp;

Each field extracted in the&nbsp;<code>Response.passport</code>&nbsp;object has a&nbsp;<code>Field.bbox</code>&nbsp;object. It&#39;s as simple list of bounding boxes relative vertices (% of the image width and height).

&nbsp;

Check out what&#39;s inside for the total_incl.bbox for example, doing:

&nbsp;

<pre>
<code>print(passport_data.passport.mrz1.bbox)</code></pre>

&nbsp;

Now that we know how to access the bounding boxes coordinates, we are going to create our&nbsp;highlighter function in which we can feed any field bounding boxes&nbsp;in different steps, as follows:

&nbsp;

<ol>
	<li>Check if the file is a pdf or an image and load it as an openCV image in both scenarios</li>
	<li>Create a mask image</li>
	<li>Loop on each feature coordinates and draw the feature rectangle on our mask</li>
	<li>Overlay the mask and original image with alpha</li>
	<li>Display image to the user</li>
</ol>

&nbsp;

Note: each coordinate returned by the API is relative (in % of the image). You&#39;ll see there is a relative to absolute conversion in the code.

&nbsp;

Here is the code step by step:

&nbsp;

<pre>
<code>def highlight_features(img_path, coordinates):
 # step 1: Check if the file is a pdf or an image and load it as an openCV image in both scenarios
 if img_path.endswith(".pdf"):
 cv_image = pdf_to_cv_image(img_path)
 else:
 cv_image = cv2.imread(img_path)

 # step 2: create mask image
 overlay = cv_image.copy()
 h, w = cv_image.shape[:2]

 # step 3: Loop on each feature coordinates and draw the feature rectangle on our mask
 for coord in coordinates:
 if len(coord):
 # Create points absolute coordinates
 pt1 = (int(w*coord[0][0]), int(h*coord[0][1]))
 pt2 = (int(w*coord[2][0]), int(h*coord[2][1]))
 cv2.rectangle(overlay, pt1, pt2, (70, 230, 244), cv2.FILLED)

 # step 4: Overlay the mask and original image with alpha
 final_image = cv2.addWeighted(overlay, 0.5, cv_image, 0.5, 0)

 # step 5: Display image to the user
 cv2.imshow("highlghted_image", cv2.resize(final_image, (600, int(600*h/w))))
 cv2.waitKey(0)</code></pre>

 
&nbsp;

Finally, we just have to modify a bit the code for&nbsp;executing the highlighting function and printing the results. Here is what the final code:

&nbsp;

&nbsp;

<pre>
<code>import cv2
import numpy as np
from pdf2image import convert_from_path
from mindee import Client


filepath = "./path/to/passport.jpg"
passport_token = "my-token-here"


def pdf_to_cv_image(filepath, page=0):
 image = convert_from_path(filepath)[page]
 return np.asarray(image)


def highlight_features(img_path, coordinates):
 # step 1: Check if the file is a pdf or an image and load it as an openCV image in both scenarios
 if img_path.endswith(".pdf"):
 cv_image = pdf_to_cv_image(img_path)
 else:
 cv_image = cv2.imread(img_path)

 # step 2: create mask image
 overlay = cv_image.copy()
 h, w = cv_image.shape[:2]

 # step 3: Loop on each feature coordinates and draw the feature rectangle on our mask
 for coord in coordinates:
 if len(coord):
 pt1 = (int(w * coord[0][0]), int(h * coord[0][1]))
 pt2 = (int(w * coord[2][0]), int(h * coord[2][1]))
 cv2.rectangle(overlay, pt1, pt2, (70, 230, 244), cv2.FILLED)

 # step 4: Overlay the mask and original image with alpha
 final_image = cv2.addWeighted(overlay, 0.5, cv_image, 0.5, 0)

 # step 5: Display image to the user
 cv2.imshow("highlghted_image", cv2.resize(final_image, (600, int(600 * h / w))))
 cv2.waitKey(0)


mindee_client = Client(passport_token=passport_token)

if __name__ == "__main__":
 response = mindee_client.parse_passport(filepath)

 print(response.passport)

 highlight_features(
 filepath,
 [
 response.passport.surname.bbox,
 response.passport.given_names[0].bbox,
 response.passport.mrz1.bbox,
 response.passport.mrz2.bbox,
 response.passport.birth_place.bbox,
 response.passport.country.bbox,
 response.passport.issuance_date.bbox,
 response.passport.expiry_date.bbox,
 response.passport.birth_date.bbox,
 response.passport.id_number.bbox
 ]
 )</code></pre>

 
&nbsp;

And the final result!

&nbsp;

<img alt="" src="https://mindee-public-website-dev.s3.amazonaws.com/blog/2020/11/04/highlghted_image.jpg" /> 
&nbsp;

&nbsp;

&nbsp;

<h2>Conclusion</h2>

&nbsp;

And there you have it! Mindee&#39;s passport API parsed a passport in under 2 seconds, matching nearly all of the sections of the passport quickly and accurately. Give it&nbsp;try with our free tier, and let us know in the chat how it works for you!

<h1>Getting started</h1>

<h2>&nbsp;</h2>

&nbsp;

Get started with passport parsing in python.

&nbsp;

<h2>1. Install the mindee python helper library</h2>

&nbsp;

Install from PyPi using&nbsp;<a href="https://pip.pypa.io/en/latest/" rel="nofollow">pip</a>, a package manager for Python.

&nbsp;

<pre>
<code class="language-bash">pip install mindee</code></pre>

&nbsp;

Don&#39;t have pip installed? Try installing it, by running this from the command line:

&nbsp;

<pre>
<code class="language-bash">$ curl https://bootstrap.pypa.io/get-pip.py | python</code></pre>

&nbsp;

Getting started with the Mindee API couldn&#39;t be easier. Create a&nbsp;<code>Client</code>&nbsp;and you&#39;re ready to go.

&nbsp;

&nbsp;

&nbsp;

<h2>2. Instantiate your Client</h2>

&nbsp;

The&nbsp;<code>mindee.Client</code>&nbsp;needs your API credentials. You can either pass these directly to the constructor (see the code below) or via environment variables.

&nbsp;

Depending on what type of document you want to parse, you need to add specifics auth token for each endpoint.

&nbsp;

<pre>
<code class="language-python">from mindee import Client

mindee_client = Client(
 passport_token="your_passport_api_token_here",
 raise_on_error=True
)</code></pre>

&nbsp;

<ul>
	<li>passport_token:&nbsp;(string) API key for passports endpoint</li>
	<li>raise_on_error: (boolean, default True) Specify wheter or not raising an Exception when HTTP errors occur</li>
</ul>

&nbsp;

&nbsp;

We suggest storing your credentials as environment variables. Why? You&#39;ll never have to worry about committing your credentials and accidentally posting them somewhere public.

&nbsp;

&nbsp;

&nbsp;

<h2>3. Parse a Passport</h2>

&nbsp;

Using the parse_passport method from your client object, you can pass any image or pdf file (see <a href="https://mindee.com/documentation/apis-overview/input-data">inputs file</a>&nbsp;for pdf pages number limits) to get the passport data.

&nbsp;

<pre>
<code class="language-python">from mindee import Client

mindee_client = Client(
 passport_token="your_passport_api_token_here",
 raise_on_error=True
)

parsed_data = mindee_client.parse_passport("/path/to/file")
</code></pre>

&nbsp;

&nbsp;

<h3>Input types</h3>

&nbsp;

You can pass your input file in three ways:

&nbsp;

From file path

<pre>
<code class="language-python">passport_data = mindee_client.parse_passport('/path/to/file', input_type="path")</code></pre>

&nbsp;

&nbsp;From a file object

<pre>
<code class="language-python">with open('/path/to/file', 'rb') as fp:
 passport_data = mindee_client.parse_passport(fp, input_type="file")
</code></pre>

&nbsp;

From a base64

<pre>
<code class="language-python">passport_data = mindee_client.parse_passport(base64_string, input_type="base64")</code></pre>

&nbsp;

&nbsp;

passport_data structure

&nbsp;

The passport_data object returned by the&nbsp;<code>parse_passport</code>&nbsp;method contains the following elements:

&nbsp;

passport_data.passport

&nbsp;

The passport attribute&nbsp;is the Passport object constructed by gathering all the pages into a single document.

&nbsp;

<pre>
<code class="language-python">passport_data.passport # returns a unique object from class Passport</code></pre>

<h4>&nbsp;</h4>

&nbsp;

passport_data.passports

&nbsp;

For multi pages pdf, the &#39;pages&#39; attribute is a list of Passport objects, each object is constructed using a unique page of the pdf

&nbsp;

<pre>
<code class="language-python">passport_data.passports # [Passport, Passport ...] </code></pre>

<h4>&nbsp;</h4>

&nbsp;

passport_data.http_response

&nbsp;

Contains the full Mindee API HTTP response object

&nbsp;

<pre>
<code class="language-python">passport_data.http_response # full HTTP request object </code></pre>

&nbsp;

&nbsp;

&nbsp;

<h2>4. Display the results</h2>

&nbsp;

You only have to print your passport object to display the different extracted fields:

&nbsp;

<pre>
<code class="language-python">from mindee import Client

mindee_client = Client(
 passport_token="your_passport_token_here",
 raise_on_error=True
)

passport_data = mindee_client.parse_passport("/path/to/passport")

print(passport_data.passport)</code></pre>

&nbsp;

&nbsp;

Get started with Passport data extraction using Mindee python library helper.

<h1>Passport fields</h1>

&nbsp;

&nbsp;

<h2>General fields structure</h2>

&nbsp;

Each Passport&nbsp;object contains a set of different fields.

&nbsp;

To acces an passport object, you need to create a <code>mindee.Client&nbsp;</code>and call the <code>Client.parse_passport</code>&nbsp;method:

&nbsp;

<pre>
<code>from mindee import Client

# Instantiate your client
mindee_client = Client(passport_token="your_token_here")

# Call the Mindee passport endpoint and parse the result
passport_data = mindee_client.parse_passport("/path/to/my/file")</code></pre>

&nbsp;

Whatever the field type, each of them contains the four following attributes:

&nbsp;

<ul>
	<li>Value: (Str or Float depending on the field type), corresponds to the field value. Set to None if the field was not extracted.</li>
	<li>Probability: (Float), confidence score of the field prediction.</li>
	<li>bbox:(Array[Float]), contains the relative vertices coordinates of the bounding box containing the field in the image. If the field is not written, the bbox is an empty array.&nbsp;</li>
	<li>reconstructed: (Bool), True if the field was reconstructed using other fields.</li>
</ul>

&nbsp;

Depending on the Field type, there can be extra attributes.

&nbsp;

&nbsp;

&nbsp;

<h2>Passport&#39;s owner data</h2>

&nbsp;

passport.given_names:&nbsp;List of passport&#39;s owner given names

<pre>
<code class="language-python"># To get the list of names
given_names = passport_data.passport.given_names

# Loop on each given name
for given_name in given_names:

 # To get the name string
 name = given_name.value</code></pre>

&nbsp;

passport.surname:&nbsp;Passport&#39;s owner surname

<pre>
<code class="language-python"># To get the passport's owner surname (string)
surname = passport_data.passport.surname.value</code></pre>

&nbsp;

passport.gender:&nbsp;Passport&#39;s owner gender (M / F)

<pre>
<code class="language-python"># To get the passport's owner gender (string among {"M", "F"}
gender = passport_data.passport.gender.value</code></pre>

&nbsp;

passport.full_name:&nbsp;Reconstructed&nbsp;Passport&#39;s owner full name from surname and given_names

<pre>
<code class="language-python"># To get the passport's owner full name (string)
full_name = passport_data.passport.full_name.value</code></pre>

&nbsp;

passport.birth_place:&nbsp;Passport&#39;s owner birth place

<pre>
<code class="language-python"># To get the passport's owner birth place (string)
birth_place = passport_data.passport.birth_place.value</code></pre>

<h2>&nbsp;</h2>

<h2>&nbsp;</h2>

<h2>Dates</h2>

&nbsp;

Each date field comes with an extra attribute:

&nbsp;

<ul>
	<li>date_object: (Datetime), datetime object from python datetime.date library</li>
</ul>

&nbsp;

&nbsp;

passport.birth_date:&nbsp;Passport&#39;s owner date of birth

<pre>
<code class="language-python"># To get the passport's owner date of birth (string)
birth_date = passport_data.passport.birth_date.value</code></pre>

&nbsp;

passport.expiry_date:&nbsp;Passport expiry date<img src="https://mindee-public-website-dev.s3.amazonaws.com/blog/2020/11/03/image-20201103134559-1.gif" />

<pre>
<code class="language-python"># To get the passport expiry date (string)
expiry_date = passport_data.passport.expiry_date.value</code></pre>

&nbsp;

passport.issuance_date:&nbsp;Passport date of issuance

<pre>
<code class="language-python"># To get the passport date of issuance (string)
issuance_date = passport_data.passport.issuance_date.value</code></pre>

&nbsp;

&nbsp;

<h2>Passports metadata</h2>

&nbsp;

passport.mrz1:&nbsp;Passport first line of machine readable zone

<pre>
<code class="language-python"># To get the passport first line of machine readable zone (string)
mrz1 = passport_data.passport.mrz1.value</code></pre>

&nbsp;

passport.mrz2:&nbsp;Passport second line of machine readable zone

<pre>
<code class="language-python"># To get the passport second line of machine readable zone (string)
mrz2 = passport_data.passport.mrz2.value</code></pre>

&nbsp;

passport.mrz: Reconstructed passport full machine readable zone from mrz1 and mrz2

<pre>
<code class="language-python"># To get the passport full machine readable zone (string)
mrz = passport_data.passport.mrz.value</code></pre>

&nbsp;

passport.id_number:&nbsp;Passport identification number

<pre>
<code class="language-python"># To get the passport id number (string)
id_number = passport_data.passport.id_number.value</code></pre>

&nbsp;

passport.country:&nbsp;Passport country code

<pre>
<code class="language-python"># To get the passport country code (string)
country_code = passport_data.passport.country_code.value</code></pre>

&nbsp;

How to use extracted passport field using Mindee python helper library.

Passport fields

<h1>Save and restore</h1>

&nbsp;

The&nbsp;<code>mindee.Client</code>&nbsp;includes methods for saving and restoring api calls. Those methods are helpful if you want to perform tests or if you simply want to reload passport&nbsp;objects corresponding to your files without performing any additional request.

&nbsp;

&nbsp;

<h2>Save a call</h2>

&nbsp;

The <code>mindee.Response</code> object returned by the <code>mindee.Client</code> when parsing a file is responsible for wrapping all the data from Mindee API calls. It&nbsp;includes a <code>dump</code> method that can be used as follows:

&nbsp;

<pre>
<code class="language-python">from mindee import Client

mindee_client = Client(passport_token="your_passport_token_here")

passport_data = mindee_client.parse_passport("./passport.jpg")

passport_data.dump("./passport_dump.json")</code></pre>

&nbsp;

&nbsp;

<h2>Restore a call</h2>

&nbsp;

To restore a client response object, you can use the <code>mindee.Response</code> as follows:

&nbsp;

<pre>
<code class="language-python">from mindee import Response

passport_data = Response.load("./passport_dump.json")</code></pre>

&nbsp;

&nbsp;

Save and restore Mindee passport API response using python helper library.

<h1>Passport data validation</h1>

&nbsp;

&nbsp;

Each Passport object comes with a set of validations checks. it can be accessed under the attribute <code>Passport.checklist</code>.

&nbsp;

Example:

&nbsp;

<pre>
<code class="language-python">from mindee import Client

mindee_client = Client(passport_token="your_passport_token_here")

passport_data = mindee_client.parse_passport("/path/to/file")

print(passport_data.passport.checklist)</code></pre>

&nbsp;

&nbsp;

This sample code should print something like:

&nbsp;

<pre>
<code class="language-bash">{
 'mrz_id_number_checksum': True, 
 'mrz_date_of_birth_checksum': True, 
 'mrz_expiration_date_checksum': True, 
 'mrz_personal_number_checksum': True, 
 'mrz_last_name_checksum': True, 
 'mrz_checksum': True
}</code></pre>

&nbsp;

&nbsp;

The machine readable zone includes many different checksums, allowing us to validate that the data it contains is correct. You can find more details&nbsp;<a href="https://en.wikipedia.org/wiki/Machine-readable_passport">here</a>.

&nbsp;

mrz_id_number_checksum: ID number in mrz checksum validation

 
mrz_date_of_birth_checksum: Date of birth in mrz checksum validation

 
mrz_expiration_date_checksum:&nbsp;Expiry date&nbsp;in mrz checksum validation

 
mrz_personal_number_checksum:&nbsp;Personnal number in mrz checksum validation

 
mrz_last_name_checksum: Last name&nbsp;in mrz checksum validation

 
mrz_checksum:&nbsp;Full mrz checksum validation

&nbsp;

Here is an&nbsp;python&nbsp;code&nbsp;for validating machine readable zone checksums:

<pre>
<code class="language-python">def check_sum(s):
 """
 https://en.wikipedia.org/wiki/Machine-readable_passport
 :param s: string
 :return: checksum value for string s
 """
 assert type(s) is str
 
 checker = 0
 alpha_to_num = {c: 10 + i for i, c in enumerate('ABCDEFGHIJKLMNOPQRSTUVWXYZ')}
 for i, c in enumerate(s):
 if i % 3 == 0:
 weight = 7
 elif i % 3 == 1:
 weight = 3
 else:
 weight = 1

 if c == '&lt;':
 val = 0
 elif c.isalpha():
 val = alpha_to_num[c]
 else:
 val = int(c)
 checker += val * weight
 return str(checker % 10)</code></pre>

&nbsp;

Passport data validation using Mindee python helper library.

Passport data validation

<h1>Performances benchmark</h1>

&nbsp;

In a few steps, you can use the mindee python library to perform benchmarks of passport extraction perfomances against your test dataset.

&nbsp;

<ol>
	<li>Perform api calls for each file in your test set</li>
	<li>Create <code>mindee.Passport</code> objects from your data (csv, json or whatever)</li>
	<li>Perform the benchmark and analyze the results</li>
</ol>

&nbsp;

&nbsp;

&nbsp;

<h2>1. Perform api calls for each file in your test set</h2>

&nbsp;

Before running the benchmark, we need to collect all the passport predictions from Mindee API and store them so we can perform the benchmark later without calling the API again.

&nbsp;

To do so, assuming that all your files are stored into a &quot;./test_dataset&quot; folder and that we want to save all the responses into a &quot;./mindee_responses&quot; folder, here is the code:

&nbsp;

<pre>
<code class="language-python">from tqdm import tqdm
from mindee import Client
import os

TEST_DATA_PATH = "./passport_data"
RESPONSES_PATH = "./passport_responses"

mindee_client = Client(passport_token="your_passport_token_here")

# Loop over all your test files
for test_filename in tqdm(os.listdir(TEST_DATA_PATH)):
 # Get the current file path
 test_file_path = os.path.join(TEST_DATA_PATH, test_filename)

 # To make sure we don't stop the process if an error occurs
 try:
 # Parse the current file
 mindee_response = mindee_client.parse_passport(test_file_path)

 # Store the response inside json file to be restored later
 # In this example we use the test file name in the json filename to
 # be able to retrieve the corresponding file
 response_filepath = os.path.join(RESPONSES_PATH, test_filename + ".json")
 mindee_response.dump(response_filepath)
 except Exception as e:
 # If an error occurs, print the filename so you can understand
 # what happened
 print(test_filename, e)</code></pre>

&nbsp;

&nbsp;

&nbsp;

<h2>2. Create <code>mindee.Passport</code> objects from your data (csv, json,&nbsp;whatever...)</h2>

&nbsp;

The <code>mindee.Passport</code> class contains a <code>compare()</code> method that takes as inputs two <code>mindee.Passport</code> objects. Before running our final script, we need now to create a&nbsp;<code>mindee.Passport object</code> containing the true labels for each fields.

&nbsp;

We&#39;ll use a csv file in this example and the&nbsp;<a href="https://pandas.pydata.org/">pandas</a>&nbsp;library.

&nbsp;

<h3>ground_truth.csv header</h3>

<pre>
<code class="language-bash">,filename,birth_date,birth_place,country,expiry_date,gender,given_names,id_number,issuance_date,mrz1,mrz2,surname
</code></pre>

<h3>&nbsp;</h3>

To construct a Passport&nbsp;object from this dummy csv example, you can simply do:

&nbsp;

<pre>
<code class="language-python">import pandas as pd
from mindee import Passport

ground_truth_df = pd.read_csv("./labels.csv")


def passport_from_csv_row(df_row):
 # For variable lengths features, split the string to get a list
 given_names = df_row["given_names"].split("|")
 return Passport(
 birth_date=df_row["birth_date"],
 birth_place=df_row["birth_place"],
 country=df_row["country"],
 expiry_date=df_row["expiry_date"],
 gender=df_row["gender"],
 given_names=given_names,
 id_number=df_row["id_number"],
 issuance_date=df_row["issuance_date"],
 mrz1=df_row["mrz1"],
 mrz2=df_row["mrz2"],
 surname=df_row["surname"]
 )


for index, df_row in ground_truth_df.iterrows():
 passport_truth = passport_from_csv_row(df_row)
 print(passport_truth)
</code></pre>

&nbsp;

&nbsp;

Running this code should print in your console something like this:

&nbsp;

<pre>
<code class="language-bash">-----Passport data-----
Filename: passport.png 
Full name: HENERT PUDARSAN 
Given names: HENERT 
Surname: PUDARSAN
Country: GBR
ID Number: 707797979
Issuance date: 2012-04-22
Birth date: 1995-05-20
Expiry date: 2017-04-22
MRZ 1: P&lt;GBRPUDARSAN&lt;&lt;HENERT&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;
MRZ 2: 7077979792GBR9505209M1704224&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;00
MRZ: P&lt;GBRPUDARSAN&lt;&lt;HENERT&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;7077979792GBR9505209M1704224&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;00
----------------------</code></pre>

&nbsp;

&nbsp;

&nbsp;

<h2>3.&nbsp;Perform the benchmark and see the results</h2>

&nbsp;

Last step, now we need to wrap it all up.

&nbsp;

The Benchmark class has two methods <code>Benchmark.add</code>&nbsp;and <code>Benchmark.save</code>&nbsp;for adding comparison between two Passport objects, and saving the final metrics.

&nbsp;

&nbsp;

<pre>
<code class="language-python">import pandas as pd
from mindee import Response, Passport, Benchmark
import os

RESPONSES_PATH = "./passport_responses"
BENCHMARK_PATH = "./benchmark_passport"

benchmark = Benchmark(BENCHMARK_PATH)
ground_truth_df = pd.read_csv("./labels.csv")


def passport_from_csv_row(df_row):
 given_names = df_row["given_names"].split("|")
 return Passport(
 birth_date=df_row["birth_date"],
 birth_place=df_row["birth_place"],
 country=df_row["country"],
 expiry_date=df_row["expiry_date"],
 gender=df_row["gender"],
 given_names=given_names,
 id_number=df_row["id_number"],
 issuance_date=df_row["issuance_date"],
 mrz1=df_row["mrz1"],
 mrz2=df_row["mrz2"],
 surname=df_row["surname"]
 )


# Loop over each file in our csv
for index, df_row in ground_truth_df.iterrows():
 try:
 # Create ground truth passport object
 ground_truth_passport = passport_from_csv_row(df_row)

 # Load the mindee Response for the current file
 mindee_response = Response.load(os.path.join(RESPONSES_PATH, df_row["filename"] + ".json"))

 # Add the comparison between the two passports to the benchmark
 benchmark.add(
 Passport.compare(mindee_response.passport, ground_truth=ground_truth_passport),
 df_row["filename"]
 )

 except Exception as e:
 print(df_row["filename"], e)

benchmark.save()</code></pre>

&nbsp;

&nbsp;

Inside our benchmark folder, you should see a new directory was created, and a metrics.png file shoud have been created inside with the different metrics:

&nbsp;

<img src="https://mindee-public-website-dev.s3.amazonaws.com/blog/2020/11/03/metrics.png" />

&nbsp;

The Passport benchmark runs on the 12 fields as shown above. For each of them, you get an information of:

accuracy: the proportion of correct predictions

precision: The proportion of correct predictions among all the non null predictions

Performances benchmark framework using Mindee python helper library for Passport OCR APIs.

Parsing Passports

<h1>&nbsp;Extracting License Plate Details</h1>

&nbsp;

After a car accident, knowing all the information about the vehicles involved is critical for insurance purposes.&nbsp; You need to have the driver&#39;s information, and information about the car.&nbsp; Imagine if you could collect it all with just your camera, and upload the photos to your adjuster automatically.

&nbsp;

Alternatively, imagine you run a trucking yard, and want to track the arrival and departures of the trucks.&nbsp; By reading the license plates - you can accurately timestamp truck movements.

&nbsp;

But a photo of a license plate is just a photo - how do we extract the information from the plates for entry in to our processes?

&nbsp;

With Mindee&#39;s License Plate API, you just upload an image of a vehicle (or vehicles for multiple plates) and the plates are identified and the text of the plate extracted.

&nbsp;

<h2>API Prerequisites</h2>

&nbsp;

<ol>
	<li>You&rsquo;ll need a free Mindee account.&nbsp;<a href="https://platform.mindee.net/signup" target="_blank">Sign up</a>&nbsp;and confirm your email to login.</li>
	<li>A photo of a car or turck witha European license plate visible. (The API is good, but not magic).</li>
</ol>

&nbsp;

<h2>Setting Up the API</h2>

&nbsp;

Log into your Mindee account and select the European License Plate API:

<img alt="" src="https://mindee-public-website-staging.s3.amazonaws.com/blog/2020/10/08/screenshot-2020-10-08-at-210825.png" />

&nbsp;

To activate the API, click the &ldquo;Try for Free&rdquo; button to access our generous free tier. You&rsquo;ll land on the dashboard page - where you can quickly see API usage (you have none right now, but that will change).&nbsp; On the left navigation, there are links to &ldquo;Documentation&rdquo;, &ldquo;Credentials&rdquo; and &ldquo;Live Interface&rdquo;.&nbsp; The docs tab has all of the technical details you&rsquo;ll need to build for the receipts API endpoint, and the Live Interface is a cool interactive demo. Rather than try out the demo, we want to build with the API,&nbsp; so click on &ldquo;Credentials&rdquo; to create an API token.

&nbsp;

Add a new token. In this example, I&rsquo;ve named it &ldquo;Tutorial&rdquo;:

&nbsp;

<img src="https://mindee-public-website.s3.amazonaws.com/blog/2020/09/01/pasted-image-0-1.png" />

&nbsp;

Click &ldquo;Add New Key&rdquo; and you&rsquo;ll be able to see your API token,.

&nbsp;

<h2>Making Your First API Call</h2>

&nbsp;

We&#39;ll use curl from the terminal for your first call.&nbsp; If you visit the Documentation tab, you can copy the code with your API key and paste directly into your command line

&nbsp;

<pre>
<code class="language-bash">curl -X POST \
 https://api.mindee.net/products/license_plates/v1/predict \
 -H 'X-Inferuser-Token: {token}' \
 -H 'content-type: multipart/form-data; boundary=----WebKitFormBoundary7MA4YWxkTrZu0gW' \
 -F file=@/Users/MyMac/Downloads/rewe_truck.jpg</code></pre>

&nbsp;

The image I am using is a truck parked in a lot:

&nbsp;

<img alt="" src="https://mindee-public-website-staging.s3.amazonaws.com/blog/2020/10/08/rewe_truck.jpg" />

<h2>&nbsp;</h2>

<h2>The Response</h2>

&nbsp;

The API response is JSON:

<pre>
<code class="language-json">{
	"call": {
		"endpoint": {
			"name": "license_plates",
			"version": "1.0"
		},
		"finished_at": "2020-10-08T20:14:26+00:00",
		"id": "58d3293f-b97b-4f5a-85ad-ed64a386f02f",
		"n_documents": 1,
		"n_inputs": 1,
		"processing_time": 0.474,
		"started_at": "2020-10-08T20:14:25+00:00"
	},
	"documents": [
		{
			"id": "22d5c578-ca4f-4b79-8071-dbd510e6db3c",
			"name": "rewe_truck.jpg"
		}
	],
	"predictions": [
		{
			"license_plates": [
				{
					"probability": 0.97,
					"segmentation": {
						"bounding_box": [
							[
								0.427,
								0.808
							],
							[
								0.515,
								0.808
							],
							[
								0.515,
								0.841
							],
							[
								0.427,
								0.841
							]
						]
					},
					"value": "BI04176"
				}
			]
		}
	]
}</code></pre>

&nbsp;

In&nbsp;under half a second, the API returned a value for the license plate &quot;BI04176&quot; with 97% confidence.&nbsp; The bounding box lists the [x,y] coordinates forming a rectangle around the plate in the image.

&nbsp;

<h2>Summary</h2>

&nbsp;

In just a few lines of code - we are able to automatically parse and extract the&nbsp;license plate information from a photo.&nbsp; There are a number of applications that can now be implemented with a connected camera, the Mindee API and a little bit of code.

&nbsp;

Let us know how you are planning to use the License plate API in your application!

Many objects in our world have unique identifiers.  For cars, the most visible and commonly used identifier is the license plate.  Every vehicle on the road has one, and they are used by police to issue tickets, identify speeders, etc.  But can we harness this power for our own applications?

Using Mindee's license plate API we can obtain the License plate simply from a photo of a vehicle.