&nbsp;&nbsp;

<h1>&nbsp;Managing errors</h1>

&nbsp;

Requests can fail for two main reasons:

&nbsp;

Network issues:&nbsp;the server could not be reached

Application error:&nbsp;the server rejected the request

&nbsp;

In the latter case, the error reported by the API&nbsp;contains:

&nbsp;

<pre>
<code class="language-json">{
 "details": "...",
 "message": "optional"
}</code></pre>

&nbsp;

&nbsp;

<table border="1" cellpadding="1" cellspacing="1">
	<thead>
		<tr>
			<th scope="col">Code</th>
			<th scope="col">Possible reasons</th>
		</tr>
	</thead>
	<tbody>
		<tr>
			<td>401 (Unauthorized)</td>
			<td>
			Bad token:&nbsp;you didn&rsquo;t provide the authentication header or the token is wrong.
			</td>
		</tr>
		<tr>
			<td>400 (Bad request)</td>
			<td>
			Wrong file type: file types allowed:&nbsp; png, jpg, webp, pdf

			&nbsp;

			Missing input file: see&nbsp;<a href="https://mindee.com/documentation/apis-overview/input-data">How to manage input files</a>&nbsp;documentation

			&nbsp;

			Too many pages in pdf:&nbsp;APIs have a maximum limit of pages allowed for pdf files

			&nbsp;

			File is too big: APIs have a maximum file size allowed
			</td>
		</tr>
		<tr>
			<td>404 (Not found)</td>
			<td>
			Wrong version: wrong version in base URL
			</td>
		</tr>
		<tr>
			<td>429&nbsp;(Too many requests)</td>
			<td>
			Plan limit reached: you have reached your maximum number of requests.

			&nbsp;

			Too high&nbsp;frequency: number of requests over time is limited. See <a href="https://mindee.com/documentation/apis-overview/authentication">Authentication documentation</a>
			</td>
		</tr>
	</tbody>
</table>

Managing errors

<h1>How to setup your Mindee account</h1>

&nbsp;

&nbsp;

Want to try Mindee APIs? You have already signed up and want to make your first API calls? You&#39;re in the right place !

&nbsp;

&nbsp;

<h2>Signup / login</h2>

&nbsp;

Go to the&nbsp;<a href="https://platform.mindee.net/signup">Signup page</a>&nbsp;to create your account. No credit card is required. You can start using the APIs right away.

&nbsp;

You already have an account ? Go to the&nbsp;<a href="https://platform.mindee.net/login">Login page</a>. If you forgot your password, click&nbsp;<a href="https://platform.mindee.net/password-forgotten">the link</a> and follow the instructions. Check your spam if you don&#39;t receive the email.

&nbsp;

&nbsp;

<h2>Subscribe to an API</h2>

&nbsp;

Once logged in, you land on our API store.

&nbsp;

<img alt="" src="https://mindee-public-website.s3.amazonaws.com/blog/2020/06/11/expense_receipt_card.jpg" />

&nbsp;

Click on the desired API card to subscribe. All our APIs have a free plan.

&nbsp;

&nbsp;

<h2>You are set to go !</h2>

&nbsp;

At this point, everything is set up to use Mindee APIs.

&nbsp;

You can either :&nbsp;

<ol>
	<li>Go to the <a href="https://mindee.com/documentation/get-started/platform-tour">Live Interface</a>&nbsp;to drag and drop images and instantly see the results</li>
	<li>Go to the <a href="https://mindee.com/documentation/get-started/platform-tour">Documentation section</a> and make your first call through the API</li>
</ol>

Setup your account

<h1>Platform tour</h1>

&nbsp;

&nbsp;

<h2>Dashboard</h2>

&nbsp;

From the dashboard section, monitor your API usage.

&nbsp;

&nbsp;

<img alt="" src="https://mindee-public-website.s3.amazonaws.com/blog/2020/06/10/dashboard_article.jpg" />

&nbsp;

Metrics :

 
Traffic: number of inputs processed (images and pdf pages).

&nbsp;

Errors:&nbsp;number of HTTP errors.

&nbsp;

Latency: average server-side computation time in seconds.

&nbsp;

Plan: your current monthly plan. Your default subscription&nbsp;is a free plan that lasts forever and automatically renews each month. Click on the &quot;change plan&quot; button to adjust your subscription according to your needs.&nbsp;While on,&nbsp;your plan will automatically be renewed each month.

&nbsp;

Quota:&nbsp;remaining number of images or pdf pages you can process under your current monthly subscription. 
&nbsp;

Invoicing: find here all your past invoices.

&nbsp;

&nbsp;

<h2>Documentation</h2>

&nbsp;

Everything you need to use the API in your apps.

&nbsp;

<img alt="" src="https://mindee-public-website-staging.s3.amazonaws.com/blog/2020/05/19/documentation_article.jpg" />

&nbsp;

HTTP request: expected HTTP request format.

&nbsp;

Features: specify the JSON response fields, data types&nbsp;and formats.

&nbsp;

Sample code: ready to use sample codes in different languages.&nbsp;Copy and paste in your environment.

&nbsp;

Usage: details about supported input files and technical limitations.

&nbsp;

JSON response: a model JSON response filled with dummy data to navigate more easily the response structure.

&nbsp;

&nbsp;

<h2>Credentials</h2>

&nbsp;

Manage your API access.

&nbsp;

<img alt="" src="https://mindee-public-website.s3.amazonaws.com/blog/2020/06/10/credentials_article.jpg" />

&nbsp;

&nbsp;

API keys : create, revoke and rename your API keys from there. Note that revokation is permanent and cannot be undone.

&nbsp;

&nbsp;

&nbsp;

<h2>Live interface</h2>

&nbsp;

This is your home for no code testing. Drag and drop your own documents to see the API in action.

&nbsp;

Quickly visualize all the extracted fields along with the raw API JSON response.

&nbsp;

<img alt="" src="https://mindee-public-website.s3.amazonaws.com/blog/2020/06/10/live_interface_article.jpg" />

Platform tour

<h1>Make your first request</h1>

&nbsp;

&nbsp;

Prerequisites:&nbsp;for the following, you&#39;ll need a Mindee account and a live API subscription. You can go learn how to create an account and freely subscribe to an API <a href="https://mindee.com/documentation/get-started/setup-your-account">here</a>.

&nbsp;

&nbsp;

<h2>Create an API key</h2>

&nbsp;

Go to the &ldquo;credentials&rdquo; section of your API. Click on &ldquo;add key&rdquo; to create a new token for this API.

&nbsp;

<img alt="" src="https://mindee-public-website.s3.amazonaws.com/blog/2020/06/10/new_token_article.jpg" />

&nbsp;

&nbsp;

&nbsp;

<h2>Get your sample code</h2>

&nbsp;

Go to the Documentation section and find the sample codes in the top right corner:

&nbsp;

<img alt="" src="https://mindee-public-website.s3.amazonaws.com/blog/2020/06/10/sample_code_article.jpg" />

&nbsp;

&nbsp;

&nbsp;

<h2>Call the API</h2>

&nbsp;

The&nbsp;sample codes will help you&nbsp;perform the POST request to Mindee APIs in different languages, and send your inputs the right way.&nbsp;

&nbsp;

For all the sample codes, replace &quot;mindee_api_url&quot; and &quot;your_token_here&quot; placeholders with your own credentials. Everything is available in your API documentation.

&nbsp;

Python

Make sure you have the <a href="https://requests.readthedocs.io/en/master/">requests</a> package installed.&nbsp;If not, you can install it in your env by running :&nbsp;

&nbsp;

<pre>
<code class="language-bash">pip install requests</code></pre>

&nbsp;

<pre>
<code class="language-python">import requests

url = "mindee_api_url"

with open("/path/to/my/file", "rb") as myfile:
 files = {"file": myfile}
 headers = {"X-Inferuser-Token": "your_token_here"}
 response = requests.post(url, files=files, headers=headers)
 print(response.text)</code></pre>

&nbsp;

&nbsp;

cURL

&nbsp;

<pre>
<code>curl -X POST \
 mindee_api_url \
 -H 'X-Inferuser-Token: your_token_here' \
 -H 'content-type: multipart/form-data; boundary=----WebKitFormBoundary7MA4YWxkTrZu0gW' \
 -F file=@/path/to/your/file.png</code></pre>

&nbsp;

&nbsp;

Javascript

Load the following code in an html file and open it with your web browser. The page displays a button allowing you to upload an image. Once uploaded, go check your console, the response is there.

&nbsp;

<pre>
<code class="language-html">&lt;form onsubmit="mindeeSubmit(event)" &gt;
 &lt;input type="file" id="my-file-input" name="file" /&gt;
 &lt;input type="submit" /&gt;
&lt;/form&gt;

&lt;script type="text/javascript"&gt;
const mindeeSubmit = (evt) =&gt; {
 evt.preventDefault()
 let myFileInput = document.getElementById('my-file-input');
 let myFile = myFileInput.files[0]
 if (!myFile) { return }
 let data = new FormData();
 data.append("file", myFile, myFile.name);

 let xhr = new XMLHttpRequest();

 xhr.addEventListener("readystatechange", function () {
 if (this.readyState === 4) {
 console.log(this.responseText);
 }
 });

 xhr.open("POST", "mindee_api_url");
 xhr.setRequestHeader("X-Inferuser-Token", "your_token_here");
 xhr.send(data);
}
&lt;/script&gt;</code></pre>

&nbsp;

You&nbsp;get something like this ;&nbsp;

&nbsp;

<img alt="" src="https://mindee-public-website.s3.amazonaws.com/blog/2020/06/10/html_article.jpg" />

&nbsp;

Ruby

&nbsp;

<pre>
<code class="language-ruby">require 'uri'
require 'net/http'
require 'net/https'
require 'mime/types'

url = URI("mindee_api_url")
file = "/path/to/your/file.png"

http = Net::HTTP.new(url.host, url.port)
http.use_ssl = true
request = Net::HTTP::Post.new(url)
request["content-type"] = 'multipart/form-data; boundary=----WebKitFormBoundary7MA4YWxkTrZu0gW'
request["X-Inferuser-Token"] = 'your_token_here'
request.body = "------WebKitFormBoundary7MA4YWxkTrZu0gW\r\nContent-Disposition: form-data; name=\"file\"; filename=\"#{File.basename(file)}\"\r\nContent-Type: #{MIME::Types.type_for(file)}\r\n\r\n------WebKitFormBoundary7MA4YWxkTrZu0gW--"

response = http.request(request)
puts response.read_body</code></pre>

&nbsp;

Node.js

Make sure you have the&nbsp;<a href="https://www.npmjs.com/package/request">request</a>&nbsp;package installed before running this code.&nbsp;If not, you can install it in your env by running :&nbsp;

&nbsp;

<pre>
<code class="language-bash">npm install request</code></pre>

&nbsp;

<pre>
<code class="language-javascript">const fs = require("fs");
const request = require("request");
const options = {
	method: "POST",
	url: "mindee_api_url",
	headers: {
		"X-Inferuser-Token": "your_token_here",
	},
	formData : {
		"file" : fs.createReadStream("/path/to/your/file.png")
	}
};
request(options, function (err, res, body) {
	if(err) console.log(err);
	console.log(body);
});</code></pre>

&nbsp;

How to make your first request to Mindee APIs

Make your first request

Get started

<h1>&nbsp;Base URL</h1>

&nbsp;

&nbsp;

Prediction APIs are served through a unique endpoint depending on the product you select. They follow this pattern :

&nbsp;

&nbsp;

<pre>
<code class="language-http">https://api.mindee.net/products/&lt;api_name&gt;/&lt;api_version&gt;/predict</code></pre>

&nbsp;

&nbsp;

You need to specify an &lt;api_name&gt; and a &lt;api_version&gt; that corresponds to the API you want to use. You can find what you need in the specific API documentation, see <a href="https://mindee.com/documentation/get-started/platform-tour" style="text-decoration:none">platform tour documentation</a> for more details.

&nbsp;

&nbsp;

<img alt="" src="https://mindee-public-website.s3.amazonaws.com/blog/2020/06/10/base_url_1_article.jpg" />

&nbsp;

<h2>About versionning</h2>

&nbsp;

We release new versions of&nbsp;our APIs for two&nbsp;reasons :&nbsp;

&nbsp;

<ul>
	<li>minor changes: &nbsp;we often release new minor changes of our APIs with new models (faster and&nbsp; / or better), without any loss of service. These releases are transparent on your side as the response structure is untouched.</li>
</ul>

&nbsp;

<ul>
	<li>major changes: this is a major iteration, with possibly no retrocompatibility of the JSON response. Those versions can include new locales, change in the way predictions are formatted... You can simultaneously use several major versions with the same subscription and tokens. You can switch between versions in the dropdown at the top of the page when inside your subscription API area.</li>
</ul>

Base URL

<h1>Authentication</h1>

&nbsp;

You must include a valid token through an HTTP custom header to authenticate your API calls :

&nbsp;

&nbsp;

<pre>
<code>'X-Inferuser-Token: &lt;my-token-here&gt;'</code></pre>

&nbsp;

&nbsp;

For each API, you can create as many tokens as you want in the credentials page. See <a href="https://mindee.com/documentation/get-started/platform-tour" style="text-decoration:none">platform tour documentation </a>for more details.

&nbsp;

&nbsp;

<h2>Token revokation</h2>

&nbsp;

&nbsp;

You can revoke a token on the credentials section. Note that once a token is revoked, you can&#39;t create a new token with the same name, and using this token to authenticate to your API will lead to a 401 error.

&nbsp;

<img alt="" src="https://mindee-public-website.s3.amazonaws.com/blog/2020/06/10/revoke_authentication_article.jpg" />

&nbsp;

<h2>Technical limitations</h2>

&nbsp;

Note that technical limitations are set up for each account depending on your plan. The default limitations are low, even for the free plans, but if you need to lower those limitations, feel free to ask the support team.

&nbsp;

<img alt="" src="https://mindee-public-website.s3.amazonaws.com/blog/2020/06/10/usage_authentication_article.jpg" />

&nbsp;

These limitations are configured for your entire subscription, regardless of tokens or IP address.

How to authenticate your requests to Mindee APIs

Authentication

<h1>Input data</h1>

&nbsp;

&nbsp;

Input files must be sent&nbsp;using&nbsp;multipart/form-data encoding with the whole binary file or stream file object, base64 are not supported yet. To see how you can make it work in your language, you can refer to&nbsp;<a href="https://mindee.com/documentation/get-started/make-your-first-request">Make your first request</a>.

&nbsp;

&nbsp;

<h2>Images</h2>

&nbsp;

We support JPG, PNG and WEBP format. Note that our APIs have a maximum file size limitation, think about downsizing your input images if you manipulate heavy files. Always prefer JPG over PDF for scans and photos.

&nbsp;

Tips for images :&nbsp;

&nbsp;

<ol>
	<li>Reduce too high resolution: downscale your image to have a faster upload time and processing time. Because the text can be very small sometimes, resizing the image too much will make it impossible to read. The rule of thumb is that big images should be resized close to 3 megapixels.</li>
	<li>Keep the aspect ratio: when resizing an image, always keep the original aspect ratio.</li>
	<li>Do not upscale: never upscale a low resolution image on your side! This will decrease the algorithm accuracy. It is best to avoid very low resolution images anyhow.</li>
	<li>Do not preprocess images: it is unnecessary to transform your image in black and white, change brightness / contrast if the image looks fine.</li>
	<li>Ask the chat!: If you have any question related to your image input data, feel free to ask the chat to talk to an expert.</li>
</ol>

&nbsp;

<h2>&nbsp;</h2>

<h2>Working with PDFs</h2>

&nbsp;

Our APIs support multi-page PDF files as well. Because we need to convert the PDF pages to actual images, the processing time may be superior compared to images processing.

&nbsp;

Tips for PDFs :&nbsp;

&nbsp;

<ol>
	<li>To be used only if you must: always prefer JPG/PNG format instead of PDF when possible. This is especially true when handling photos and scans. PDF files should be reserved for digital documents only for maximum performance.</li>
	<li>Working with several pages: &nbsp;your PDF may contain several pages. In this scenario, we will handle each page as a separate document. You will therefore receive as many predictions as there are pages. See <a href="https://mindee.com/documentation/apis-overview/json-response-scheme">JSON response scheme</a> documentation for more details.</li>
	<li>Limitations: here is a maximum number of pages you can send, check the Documentation page of your selected API.</li>
</ol>

Input data best practices for Mindee APIs

Input data

<h1>JSON response scheme</h1>

&nbsp;

Each API response follows the same response scheme. The predictions content is specific with the API you use. Once you have subscribed to your preferred API, read the documentation to check for all the outputs available.

&nbsp;

<pre>
<code class="language-json">{
 "call": {
 "endpoint": {
 "name": "endpoint", 
 "version": "2.1"
 },
 "finished_at": "2017-04-11T02:30:49+00:00", 
 "id": "2a81a403-0389-4b39-9b1f-22b5febf795e", 
 "n_documents": 1, 
 "n_inputs": 1, 
 "processing_time": 0.725, 
 "started_at": "2017-04-11T02:30:49+00:00"
 }, 
 "documents": [
 {
 "id": "bf53e6b1-1d29-4187-97ec-bab10eca5744", 
 "name": "filename.png"
 }
 ],
 "predictions": [

 ...

 ]
}
</code></pre>

&nbsp;

call:&nbsp;metadata about the request

&nbsp;

id: a UUID - this identifies your http call

processing_time: server side processing time in seconds

started_at: datetime for when the call has been received

finished_at: datetime the call has been fully processed

n_documents:&nbsp; number of documents processed.

n_inputs: number of images and/or pdf pages processed. Your quota will be decreased by this value.

endpoint:

&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; name: name of the endpoint used to make the call

&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; version: selected API version

&nbsp;

&nbsp;

documents&nbsp;:&nbsp;array with the details of each document sent to the API. NB: a pdf page counts for one document.

id: a UUID - this identifies your document

name: name of file sent, for a pdf this may add the page number at the end

&nbsp;

predictions: Array with the predictions for each document processed. The structure inside each element of this array is specific with the API you use. NB: for now, each pdf page will output a prediction

General JSON response scheme for Mindee APIs

JSON response scheme

APIs overview

<h1>Parsing receipts with Mindee&#39;s API</h1>

&nbsp;

Anyone who has filed an expense report can tell you: receipt tracking and expense logging is a headache.&nbsp; Enter Mindee&rsquo;s receipt parsing API, which uses deep learning to automatically, accurately and instantaneously parse your receipt details.

&nbsp;

In this tutorial, we will walk through the steps to use Mindee&rsquo;s Receipt Parsing API.&nbsp; Let&rsquo;s get started!&nbsp;

&nbsp;

<h2>API Prerequisites</h2>

<ol>
	<li>You&rsquo;ll need a free Mindee account. <a href="https://platform.mindee.net/signup" target="_blank">Sign up</a>&nbsp;and confirm your email to login.</li>
	<li>A receipt.&nbsp; Look in your bag/wallet for a recent one, or do a <a href="https://www.google.com/search?q=cafe+receipt" target="_blank">Google Image search</a> for a receipt and download a few to test with.</li>
</ol>

&nbsp;

&nbsp;

&nbsp;<img src="https://lh3.googleusercontent.com/Dlw1124fbpDBiZamK8ohelvR9MNK0UlnKutB9UDTfp4vD6FzNN26sXwlkJUFGhiTNvAW_PxWNFz98dGSicLm8qkrEey7VVBY1AE_bErDOKLILhnFGL3UzEmvVetBfad-70qmSXLG" />

&nbsp;

<h2>Setting up the API</h2>

&nbsp;

Log into you Mindee account and access your Expense Receipt API environment by clicking the Expense Receipts card:

&nbsp;

<img src="https://lh6.googleusercontent.com/7V0dnGLFfrvOHViat-kxpTs888k2C3Nk1AJlSlDGks9U5tJD2DjWXXMzC0ks-BcmfRR4-gfKILmhtLesNoMrdfRBjb6OmySZALeqeuUNTBEDnLc_87Kc-vkRYbVKhGjmh4bTKQx9" />

&nbsp;

To activate the API, click the &ldquo;Try for Free&rdquo; button to access our generous free tier. You&rsquo;ll land on the dashboard page - where you can quickly see API usage (you have none right now, but that will change).&nbsp; On the left navigation, there are links to &ldquo;Documentation&rdquo;, &ldquo;Credentials&rdquo; and &ldquo;Live Interface&rdquo;.&nbsp; The docs tab has all of the technical details you&rsquo;ll need to build for the receipts API endpoint, and the Live Interface is a cool interactive demo. Rather than try out the demo, we want to build with the API,&nbsp; so click on &ldquo;Credentials&rdquo; to create an API token.

&nbsp;

Add a new token. In this example, I&rsquo;ve named it &ldquo;Tutorial&rdquo;:

&nbsp;

<img src="https://mindee-public-website.s3.amazonaws.com/blog/2020/09/01/pasted-image-0-1.png" style="height:247px; width:500px" />

&nbsp;

Click &ldquo;Add New Key&rdquo; and you&rsquo;ll be able to see your API token,.

&nbsp;

Now, we are ready to make an API call.&nbsp; In this example, we&rsquo;ll be using cURL.

&nbsp;

<pre>
<code class="language-bash">curl -X POST \ https://api.mindee.net/products/expense_receipts/v2/predict \ 

-H 'X-Inferuser-Token: {apiToken}’ \

-H 'content-type: multipart/form-data; boundary=----WebKitFormBoundary7MA4YWxkTrZu0gW' \

-F file=@/path/to/your/file.png</code></pre>

&nbsp;

Simply replace {apiToken}&nbsp;with your new API token and /path/to/your/file/png with the path to your receipt.&nbsp;

NOTE: You can also copy this code right from the documentation tab of the API with your API token inserted for you.

&nbsp;

In this example, I used a receipt from the grocery store in Koln airport (my last business trip in 2020):

&nbsp;

<img src="https://mindee-public-website.s3.amazonaws.com/blog/2020/09/03/comporessed_doug_receipt.jpg" style="height:717px; width:500px" />

&nbsp;

Pasting the cURL sample into my terminal, I hit enter and about a second later, I received a JSON response with the receipt details.&nbsp; The full JSON can be accessed <a href="https://mindee-public-website.s3.amazonaws.com/blog/2020/09/01/rewe.json">here</a>.&nbsp; Since the response is quite verbose, we will walk through the various fields section by section.

&nbsp;

&nbsp;

<h2>API Response: Parsing Results</h2>

&nbsp;

<h3>Summary &amp; Documents section:</h3>

&nbsp;

The first two sections of the response contain information about the API call made:

&nbsp;

<pre>
<code class="language-json">"call": {
    "endpoint": {
        "name": "expense_receipts",
        "version": "2.1"
    },
    "finished_at": "2020-08-29T18:01:22+00:00",
    "id": "e47d8654-0df7-4839-a282-2c04bf293886",
    "n_documents": 1,
    "n_inputs": 1,
    "processing_time": 1.087,
    "started_at": "2020-08-29T18:01:21+00:00"
},
"documents": [
    {
    "id": "66d9adc6-76cf-4c42-8622-3dadb660ac32",
    "name": "IMG_20200301_073354.jpg"
    }
],</code></pre>

&nbsp;

The call section tells us that we ran on the expense receipts endpoint, uploading one document that is one page long, and after just about 1 second, the file was processed.&nbsp; The documents section gives the Mindee id for the file, and the filename.

&nbsp;

<h2>Predictions:&nbsp;&nbsp;</h2>

&nbsp;

Now we are getting to the exciting stuff.&nbsp; The Predictions section is broken into several sections.&nbsp; Several of these are identifying fields on the receipt, and others are using Machine Learning to deduce information from the receipt.&nbsp; Let&rsquo;s go through each section:

<h3>&nbsp;</h3>

<h3>Category</h3>

&nbsp;

<pre>
<code class="language-json">category": {
     "probability": 0.51,
     "value": "miscellaneous"
},</code></pre>

&nbsp;

The API make a prediction on the type of purchase.&nbsp; In this case, it is 51% sure it is miscellaneous.&nbsp; The possible categories are [toll, food, parking, transport, accommodation, gasoline, miscellaneous].

&nbsp;

<h3>Date</h3>

&nbsp;

Identified from text on&nbsp;the receipt and converted into ISO format. This purchase was made on February 3, 2020, and the model is 99% confident in that choice.&nbsp; The segmentation bounding box provides 4 (x,y) coordinates indicating where the date was pulled from the receipt [(0,0) is the upper left corner, (1,1) is the bottom right corner].

&nbsp;

<pre>
<code class="language-json">"date": {
    "iso": "2020-02-03",
    "probability": 0.99,
    "raw": "03-02-2020",
    "segmentation": {
    "bounding_box": [
      [0.64,0.661],
      [0.801,0.661],
      [0.801,0.686],
      [0.64,0.686]
  ]
}</code></pre>

&nbsp;

<h3>Locale</h3>

&nbsp;

Using data from the&nbsp;receipt, the API can predict where the purchase was made, the language and the currency:. Check the documentation for the latest support. At the time of writing, support is centered on Europe and North America.

&nbsp;

<pre>
<code class="language-json">"locale": {
    "country": "DE",
    "currency": "EUR",
    "language": "de",
    "probability": 0.77,
    "value": "de-DE"
},</code></pre>

&nbsp;

In the case of my receipt, it is 77% confident that the purchase is in German, made in Germany, and in euros.

&nbsp;

<h3>Merchant</h3>

&nbsp;

<pre>
<code class="language-json">"merchant": {
    "name": "REWE",
    "probability": 0.91,
    "segmentation": {
    "bounding_box": [
      [0.279, 0.135],
      [0.719, 0.135],
      [0.719,0.23],
      [0.279,0.23]
     ]
}</code></pre>

The API correctly predicted (it was 91% sure) that it was a REWE store. Again, four (x,y) points mark the location of the text naming the Merchant on the image.

&nbsp;

<h3>Orientation</h3>

&nbsp;

<pre>
<code class="language-json">"orientation": {
   "degrees":0,
   "probability": 0.99
},</code></pre>

Did the document require rotation before parsing?&nbsp; Measured in 90 degree increments [0.90.180.270]. In&nbsp;this case, it did not require any rotation.

&nbsp;

<h3>Taxes</h3>

&nbsp;

<pre>
<code class="language-json">"taxes": [],</code></pre>

&nbsp;

If any taxes are identified in the receipt, the will appear here.&nbsp; In this case, no taxes&nbsp;were found (but this is the correct result).

<h2>&nbsp;</h2>

<h3>Time</h3>

&nbsp;

<pre>
<code class="language-json">"time": {
    "iso": "15:50",
    "probability": 0.99,
    "raw": "15:50",
    "segmentation": {
    "bounding_box": [
      [0.649,0.898],
      [0.732,0.898],
      [0.732,0.925],
      [0.649,0.925]
    ]
}</code></pre>

&nbsp;

Time the receipt was printed, confidence, and the (x,y) coordinates that bound the field in the image.

<h2>&nbsp;</h2>

<h3>Total</h3>

&nbsp;

Perhaps the most important part of the receipt, the total spent, along with confidence and the box indicating the location on the receipt.

&nbsp;

<pre>
<code class="language-json">"total": {
    "amount": 17.74,
    "probability": 0.99,
    "segmentation": {
    "bounding_box": [
     [0.663,0.589],
     [0.765,0.589],
     [0.765,0.617],
     [0.663,0.617]
   ]
}</code></pre>

&nbsp;

<h2>&nbsp;</h2>

<h2>Summary:</h2>

&nbsp;

In just over 1 second, a receipt was uploaded, parsed and the response returned to the end user.&nbsp; We know that &euro;17.74 was spent at a REWE in Germany on February 3, 2020 at 15:50.&nbsp; Using the bounding boxes, we can have the user validate the values, and then input this data into an expense management system.&nbsp;&nbsp;

&nbsp;

&nbsp;

<h2>Conclusion</h2>

&nbsp;

Using the MIndee receipt parsing API, you can quickly validate receipts, allowing for faster, more accurate (and less painful) expense management for our users. If you have questions, please reach out to us in the chat widget in the bottom right.

Anyone who has filed an expense report can tell you: receipt tracking and expense logging is a headache.  Enter Mindee’s receipt parsing API, which uses deep learning to automatically, accurately and instantaneously parse your receipt details. In this tutorial, we will walk through the steps to use Mindee’s Receipt Parsing API.  Let’s get started!

<h1>Extract receipt data with Mindee&#39;s API using python</h1>

&nbsp;

In this tutorial, you will learn how to parse a receipt image using python. This can help you automatically extract&nbsp;data from receipts without human interaction and use it in your backend applications for many accounting purposes.

&nbsp;

The final script can be used within any python middleware application or REST API serving your frontend interfaces.

&nbsp;

In the end of this tutorial, we&rsquo;ll show you how to highlight the extracted receipt fields on the image using the open source computer vision library OpenCV.

&nbsp;

The final result looks like this:

&nbsp;

<img alt="" src="https://mindee-public-website-staging.s3.amazonaws.com/blog/2020/09/03/ezgifcom-video-to-gif-14.gif" />

&nbsp;

&nbsp;

<h2>API Prerequisites</h2>

&nbsp;

<ol>
	<li>You need to have python installed in your environement. We&#39;re using the 3.7 version in this tutorial.</li>
	<li>You&rsquo;ll need a free Mindee account. <a href="https://platform.mindee.net/signup" target="_blank">Sign up</a>&nbsp;and confirm your email to login.</li>
	<li>A receipt.&nbsp; Look in your bag/wallet for a recent one, or do a <a href="https://www.google.com/search?q=cafe+receipt" target="_blank">Google Image search</a> for a receipt and download a few to test with.</li>
</ol>

&nbsp;

<h2>&nbsp;</h2>

<h2>Setup the project</h2>

&nbsp;

Create an empty directory wherever you want in your laptop, we&rsquo;ll call ours &ldquo;python_receipt&rsquo;, and create a &lsquo;main.py&rsquo; file in it.

&nbsp;

Open up&nbsp;a terminal and go to the directory you just created. In this tutorial we&rsquo;ll use the python library &ldquo;requests&rdquo; to call the API. If it&rsquo;s not installed in your env, you can install by running:

&nbsp;

<pre>
<code class="language-bash">pip install requests</code></pre>

&nbsp;

Make sure you have a receipt image somewhere on your laptop, and you know the path.

&nbsp;

That&rsquo;s it for the setup, let&rsquo;s call the API.

&nbsp;

&nbsp;

<h2>Call the Mindee receipt API</h2>

&nbsp;

Login onto the platform.mindee.net and get into your Expense receipt API environment by clicking the following card.

&nbsp;

<img alt="Expense receipt API card" src="https://mindee-public-website-staging.s3.amazonaws.com/blog/2020/09/03/expense_receipt_card.jpg" />

&nbsp;

If you don&rsquo;t have an API token yet for this API, go to the &ldquo;credentials&rdquo; section and create a new API Key.&nbsp;

&nbsp;

<img alt="new api token" src="https://mindee-public-website-staging.s3.amazonaws.com/blog/2020/09/03/new_token.jpg" />

&nbsp;

Click on the documentation section in the navbar, and then click on the Python link on the sample code area. Copy the code.

&nbsp;

<img alt="" src="https://mindee-public-website-staging.s3.amazonaws.com/blog/2020/09/03/python_sample.jpg" />

&nbsp;

Open the &ldquo;main.py&rdquo; file you created a few minutes ago, and paste your sample code, it should looks like:

&nbsp;

<pre>
<code class="language-ruby">import requests 

url = "https://api.mindee.net/products/expense_receipts/v2/predict" 

with open("/path/to/my/file", "rb") as myfile: 
 files = {"file": myfile} 
 headers = {"X-Inferuser-Token": "my-token-here"} 
 response = requests.post(url, files=files, headers=headers) 
 print(response.text)</code></pre>

&nbsp;

&nbsp;

Replace the &ldquo;path/to/my/file&rdquo; placeholder in the code with the path of the receipt image you want to extract data from.

&nbsp;

Replace the &ldquo;my-token-here&rdquo; with the API token you created previously in the platform.

&nbsp;

You can get back to your console and run the script

&nbsp;

<pre>
<code class="language-python">python main.py</code></pre>

&nbsp;

You should see a json object&nbsp;in your console with the results extracted from the receipt parsing API.&nbsp;Now let&rsquo;s parse this json to get the data we need.

&nbsp;

<h2>&nbsp;</h2>

<h2>Parse the result</h2>

&nbsp;

The API extract a list of different fields in the receipt (total amount, taxes, date &hellip;). You can find a description of all o fthe fields in the &ldquo;documentation&rdquo; tab on&nbsp;Mindee API platform page.

&nbsp;

Now that we coded the API request, we&rsquo;ll use the json object and get a few extracted features.

&nbsp;

First, let&rsquo;s create a parsing function that returns the receipt total amount, the date and hour of the payment&nbsp;and the expense category. We&rsquo;ll focus only on those four features for the tutorial.

&nbsp;

The features are inside the prediction object in the json response. As you can send a multi pages pdf to the API, the predictions object is an array, with a prediction object for each page.

&nbsp;

In our case, as we are sending a simple jpg image, the predictions array contains only one object.&nbsp;

&nbsp;

<pre>
<code class="language-python">def get_features(json_response):
 parsed_data = {}

 prediction = json_response["predictions"][0]

 parsed_data["total_amount"] = prediction["total"]["amount"]
 parsed_data["time"] = prediction["time"]["iso"]
 parsed_data["date"] = prediction["date"]["iso"]
 parsed_data["category"] = prediction["category"]["value"]

 return parsed_data</code></pre>

&nbsp;

Now that our function is ready, we&rsquo;ll just print the result in the console and add a simple check to ensure the API call was well performed and we can access the result. You can replace the main.py script to this:

&nbsp;

<pre>
<code class="language-ruby">import requests


def get_features(json_response):
 parsed_data = {}

 prediction = json_response["predictions"][0]

 parsed_data["total_amount"] = prediction["total"]["amount"]
 parsed_data["time"] = prediction["time"]["iso"]
 parsed_data["date"] = prediction["date"]["iso"]
 parsed_data["category"] = prediction["category"]["value"]

 return parsed_data


url = "https://api.mindee.net/products/expense_receipts/v2/predict"

with open("/path/to/my/file", "rb") as myfile:
 files = {"file": myfile}
 headers = {"X-Inferuser-Token": "my-token-here"}
 response = requests.post(url, files=files, headers=headers)

 if response.status_code != 200:
 print("Request error")
 else:
 json_response = response.json()
 features = get_features(json_response)
 print("Date:", features["date"])
 print("Time:", features["time"])
 print("Total amount:", features["total_amount"])
 print("Category:", features["category"])</code></pre>

&nbsp;

Run the script and check out the results on your image.

&nbsp;

Finally we are going to add a last piece of code that will highlight the features extracted on the image. It can help you or your users to very quickly validate the data extraction.

&nbsp;

&nbsp;

<h2>Highlight features on the image</h2>

&nbsp;

Let&rsquo;s try to highlight the features as if someone did it with a pen.

&nbsp;

First, you&rsquo;ll need to install the computer vision python library OpenCV if you don&rsquo;t have it already installed in your env. To do so, run:

&nbsp;

<pre>
<code class="language-bash">pip install opencv-python</code></pre>

&nbsp;

We need to change a bit our get_features function to return coordinates for each feature, as we&#39;re using this to know where to draw the rectangles.

&nbsp;

<pre>
<code class="language-python">def get_features(json_response):
 parsed_data = {}
 coordinates = []

 prediction = json_response["predictions"][0]

 parsed_data["total_amount"] = prediction["total"]["amount"]
 parsed_data["time"] = prediction["time"]["iso"]
 parsed_data["date"] = prediction["date"]["iso"]
 parsed_data["category"] = prediction["category"]["value"]

 coordinates.append(prediction["total"]["segmentation"]["bounding_box"])
 coordinates.append(prediction["time"]["segmentation"]["bounding_box"])
 coordinates.append(prediction["date"]["segmentation"]["bounding_box"])

 return parsed_data, coordinates
</code></pre>

&nbsp;

Once it&rsquo;s done, we are going to create our highlighter function in different steps, as follows:

<ol>
	<li>Open the image from path</li>
	<li>Create a mask image</li>
	<li>Loop on each feature coordinates and draw the feature rectangle on our mask</li>
	<li>Overlay the mask and original image with alpha</li>
	<li>Display image to the user</li>
</ol>

&nbsp;

Note: each coordinate returned by the API is relative (in % of the image). You&#39;ll see there is a relative to absolute conversion in the code.

&nbsp;

Here is the code step by step:

&nbsp;

<pre>
<code class="language-ruby">def highlight_features(img_path, coordinates):
 # step 1: Open the image from path
 cv_image = cv2.imread(img_path)

 # step 2: create mask image
 overlay = cv_image.copy()

 h, w = cv_image.shape[:2]

 # step 3: Loop on each feature coordinates and draw the feature rectangle on our mask
 for coord in coordinates:
 pt1 = (int(w*coord[0][0]), int(h*coord[0][1]))
 pt2 = (int(w*coord[2][0]), int(h*coord[2][1]))
 cv2.rectangle(overlay, pt1, pt2, (70, 230, 244), cv2.FILLED)

 # step 4: Overlay the mask and original image with alpha
 final_image = cv2.addWeighted(overlay, 0.5, cv_image, 0.5, 0)

 # step 5: Display image to the user
 cv2.imshow("highlghted_image", cv2.resize(final_image, (400, int(400*h/w))))
 cv2.waitKey(0)</code></pre>

&nbsp;

Finally, we just have to modify a bit the code for executing the highlighting function and printing the results. Here is what the final code looks like:

&nbsp;

<pre>
<code class="language-ruby">import requests
import cv2



def get_features(json_response):
 parsed_data = {}
 coordinates = []

 prediction = json_response["predictions"][0]

 parsed_data["total_amount"] = prediction["total"]["amount"]
 parsed_data["time"] = prediction["time"]["iso"]
 parsed_data["date"] = prediction["date"]["iso"]
 parsed_data["category"] = prediction["category"]["value"]

 coordinates.append(prediction["total"]["segmentation"]["bounding_box"])
 coordinates.append(prediction["time"]["segmentation"]["bounding_box"])
 coordinates.append(prediction["date"]["segmentation"]["bounding_box"])

 return parsed_data, coordinates


def highlight_features(img_path, coordinates):
 # step 1: Open the image from path
 cv_image = cv2.imread(img_path)

 # step 2: create mask image
 overlay = cv_image.copy()

 h, w = cv_image.shape[:2]

 # step 3: Loop on each feature coordinates and draw the feature rectangle on our mask
 for coord in coordinates:
 pt1 = (int(w*coord[0][0]), int(h*coord[0][1]))
 pt2 = (int(w*coord[2][0]), int(h*coord[2][1]))
 cv2.rectangle(overlay, pt1, pt2, (70, 230, 244), cv2.FILLED)

 # step 4: Overlay the mask and original image with alpha
 final_image = cv2.addWeighted(overlay, 0.5, cv_image, 0.5, 0)

 # step 5: Display image to the user
 cv2.imshow("highlghted_image", cv2.resize(final_image, (400, int(400*h/w))))
 cv2.waitKey(0)


url = "https://api.mindee.net/products/expense_receipts/v2/predict"

with open("/path/to/my/file", "rb") as myfile:
 files = {"file": myfile}
 headers = {"X-Inferuser-Token": "my-token-here"}
 response = requests.post(url, files=files, headers=headers)

 if response.status_code != 200:
 print("Request error")
 else:
 json_response = response.json()
 features, coords = get_features(json_response)
 print("Date:", features["date"])
 print("Time:", features["time"])
 print("Total amount:", features["total_amount"])
 print("Category:", features["category"])
 highlight_features("path/to/my/file", coords)
</code></pre>

&nbsp;

In the last line of code, don&#39;t forget to replace the placeholder &quot;path/to/my/file&quot; to your current image path.

And&nbsp;the final result!

&nbsp;

<img alt="" src="https://mindee-public-website-staging.s3.amazonaws.com/blog/2020/09/03/final_result_python_receipt.jpg" />

&nbsp;

&nbsp;

<h2>Conclusion</h2>

&nbsp;

In just over 1 second, a receipt was uploaded, the API extracted the receipt data, then the result was&nbsp;parsed&nbsp;and&nbsp;highlighted in&nbsp;the image. Cool right?

&nbsp;

If you wanted to use this kind of script to display results to your users, i&#39;d advice you to do the highlighting in the front-end application, as sending images back from your middleware is not the best option because of payload sizes. The other solution would be to store the final image using cv2.imwrite(...) but it would make your client download the result. To help you with client-side rendering for your images, you can check out our&nbsp;<a href="https://github.com/publicMindee/react-mindee-js">open source ReactJS SDK</a>.

&nbsp;

If you have questions, please reach out to us in the chat widget in the bottom right.

This tutorial explains how to extract data from receipt image using python and Mindee"s API, then how to parse the result and highlight the different features on the source image.

python

<h1>Extract receipt data with Mindee&rsquo;s API using reactjs</h1>

&nbsp;

In this tutorial, you&#39;ll learn how to create an optimized and interactive user interface for automatic receipt data extraction using ReactJS and Mindee. We are going to use both Mindee API and open source <a href="https://github.com/publicMindee/react-mindee-js">ReactJS SDK</a>. The SDK has a lot of features built-in such as shapes visualisation and interactions, zoom, multi pages ... It will help&nbsp;you create any dynamic user experience based on the receipt image&nbsp;to help your users validate very quickly the results predicted by the API.

&nbsp;

In 10 minutes,&nbsp;you&#39;ll get something like this:

&nbsp;

<img alt="Mindee receipt parsing react js" src="https://mindee-public-website-staging.s3.amazonaws.com/blog/2020/09/04/ezgifcom-optimize-2.gif" />

&nbsp;

Let&#39;s get started!

&nbsp;

<h2>&nbsp;</h2>

<h2>API Prerequisites</h2>

&nbsp;

<ol>
	<li>We are using create-react-app framework for bootstrapping the app. If you don&#39;t have it installed, you can do it&nbsp;<a href="https://create-react-app.dev/docs/getting-started/">here</a>. You can also use another React framework for creating the app but some code may differ from what is written in this tutorial.</li>
	<li>You&rsquo;ll need a free Mindee account. <a href="https://platform.mindee.net/signup" target="_blank">Sign up</a>&nbsp;and confirm your email to login.</li>
	<li>A receipt.&nbsp; Look in your bag/wallet for a recent one, or do a <a href="https://www.google.com/search?q=cafe+receipt" target="_blank">Google Image search</a> for a receipt and download a few to test with.</li>
</ol>

&nbsp;

&nbsp;

<h2>Setup your project</h2>

&nbsp;

First, we need to create a new react application using create-react-app. Open up your terminal and go to the directory you want to create your project in, and create your app with the name you want (we call ours receipt_parsing):

&nbsp;

<pre>
<code class="language-bash">npx create-react-app receipt_parsing</code></pre>

&nbsp;

To make sure everything went well, go to your directory and run the app

&nbsp;

<pre>
<code class="language-bash">cd receipt_parsing
yarn start</code></pre>

&nbsp;

Or using npm

&nbsp;

<pre>
<code class="language-bash">cd receipt_parsing
npm run start</code></pre>

&nbsp;

Your browser is going to open a new window and run the app.

&nbsp;

A last package install before we can start coding! As we are going to use Mindee ReactJS SDK, we need to install it. It&#39;s open-source and you can find all the documentation on the <a href="https://github.com/publicMindee/react-mindee-js">Github repo</a>. To install in your React app, just run in your terminal:

&nbsp;

<pre>
<code class="language-bash">yarn add react-mindee-js</code></pre>

&nbsp;

Or using npm:

&nbsp;

<pre>
<code class="language-bash">npm install react-mindee-js</code></pre>

&nbsp;

Now, using your favourite IDE, open the App.js file that was created by create-react-app, inside the src folder of your project directory. In the whole tutorial, we are going to work only on this file. Once you access the App.js file in your terminal, delete everything and just paste this code:

&nbsp;

<pre>
<code class="language-ruby">import React, {useState} from 'react';

import {
 AnnotationViewer,
 createPolygonFromCoordinates
} from "react-mindee-js";

const HelloWorld = () =&gt; {
 const [image, setImage] = useState(null)

 const shapes = [
 {
 polygon: createPolygonFromCoordinates([[0.66, 0.3], [0.12, 0.5], [0.52, 0.8], [0.82, 0.24]]),
 color: "#fd3246",
 data:"This is red shape",
 index: 0,
 active: {
 color: '#00f0f0',
 lineWidth: 3
 }
 }
 ]

 return (
 &lt;div&gt;
 &lt;input type="file" onChange={e =&gt; setImage(URL.createObjectURL(e.target.files[0]))}/&gt;
 {
 image &amp;&amp; &lt;div style={{ height: '500px', width: '400px'}}&gt;
 &lt;AnnotationViewer
 shapes={shapes}
 image={image}
 onShapeClick={shape =&gt; console.log('shapeClicked', shape)}
 onShapeHover={shape =&gt; console.log('shapeHovered', shape)}
 /&gt;
 &lt;/div&gt;
 }
 &lt;/div&gt;
 );
}
export default HelloWorld;</code></pre>

&nbsp;

We define a polygon by its [x,y] coordinates and colors.&nbsp; When an image is added from the form, we call the onChange to apply the polygon, and write to the console on Click and Hover actions.

You&#39;re all setup to start. When you run the app, you should see a&nbsp;file input button on your web page, and once you upload an image using it, you get something like:

&nbsp;

<img alt="" src="https://mindee-public-website-staging.s3.amazonaws.com/blog/2020/09/04/hello_world_receipt_react.jpg" />

The ploygon is defined but the four [x,y] vertices, and the color is red, but on hover, it turns blue (00f0f0).

<h2>&nbsp;</h2>

<h2>&nbsp;</h2>

<h2>Parse the receipt</h2>

&nbsp;

Log&nbsp;onto the <a href="https://platform.mindee.net/login">platform</a> and get into your Expense receipt API environment by clicking the following card.

<img alt="Expense receipt API card" src="https://mindee-public-website-staging.s3.amazonaws.com/blog/2020/09/03/expense_receipt_card.jpg" />

&nbsp;

If you don&rsquo;t have an API token yet for this API, go to the &ldquo;credentials&rdquo; section and create a new API Key.&nbsp;

&nbsp;

<img alt="new api token" src="https://mindee-public-website-staging.s3.amazonaws.com/blog/2020/09/03/new_token.jpg" />

&nbsp;

Inside your App.js file, replace everything with this code, and replace the &#39;my-token-here&#39; placeholder with the one you just created on the platform:

&nbsp;

<pre>
<code class="language-javascript">import React, {useState} from 'react';
import {
 AnnotationViewer,
 formatPrediction
} from "react-mindee-js";

const HelloWorld = () =&gt; {
 const [image, setImage] = useState(null)
 const [shapes, setShapes] = useState([])

 const parseReceipt = (e) =&gt; {
 let myFile = e.target.files[0]

 if (!myFile) {return}
 
 setImage(URL.createObjectURL(myFile))

 let data = new FormData();
 data.append("file", myFile, myFile.name);

 let xhr = new XMLHttpRequest();

 xhr.addEventListener("readystatechange", function () {
 if (this.readyState === 4) {
 let json_response = JSON.parse(this.responseText)
 setShapes(formatPrediction(json_response.predictions[0]))
 }
 });

 xhr.open("POST", "https://api.mindee.net/products/expense_receipts/v2/predict");
 xhr.setRequestHeader("X-Inferuser-Token", "my-token-here");
 xhr.send(data);
 }

 return (
 &lt;div&gt;
 &lt;input type="file" onChange={e =&gt; parseReceipt(e)}/&gt;
 {
 image &amp;&amp; &lt;div style={{ height: '500px', width: '400px'}}&gt;
 &lt;AnnotationViewer
 shapes={shapes}
 image={image}
 onShapeClick={shape =&gt; console.log('shapeClicked', shape)}
 onShapeHover={shape =&gt; console.log('shapeHovered', shape)}
 /&gt;
 &lt;/div&gt;
 }
 &lt;/div&gt;
 );
}
export default HelloWorld;</code></pre>

&nbsp;

In this iteration, we have replaced the onChange callback function to&nbsp;a new function parseReceipt. Basically, this function will call the Receipt parsing API (and send the image to Mindee to extract the receipt details), and transform the results to very easily generate shapes using formatPrediction() function from the SDK.

&nbsp;

Run the app, you should see something like this when you upload a receipt image:

&nbsp;

<img alt="react mindee parse receipt" src="https://mindee-public-website-staging.s3.amazonaws.com/blog/2020/09/04/hello_world_receipt_react_step.jpg" />

&nbsp;

All boxes displayed on the image are interractive and you can bind custom onClick and onMouseOver mouse events functions to them. You can also try to zoom in the image, try to move the image inside the viewer etc.. All features are explained on the Github repo.

&nbsp;

Important note: Calling the receipt parsing API client-side is not&nbsp;secure, as your API token is present in the browser data, potentially exposing this to the world.&nbsp; This approach should only be used for&nbsp;prototyping,&nbsp;you&#39;ll need a back-end middleware to make a secure production app.

&nbsp;

Now we are just going to change a few lines of codes to display the data extracted by the API. To do so, we are going to add a new state object &#39;receiptData&#39;, and display a few features on the web page. In your code, change the HelloWorld component to this (don&#39;t forget to fix your api token):

&nbsp;

<pre>
<code class="language-javascript">const HelloWorld = () =&gt; {
 const [image, setImage] = useState(null)
 const [shapes, setShapes] = useState([])
 const [receiptData, setReceiptData] = useState(null)

 const parseReceipt = (e) =&gt; {
 let myFile = e.target.files[0]

 if (!myFile) {
 return
 }
 setImage(URL.createObjectURL(myFile))

 let data = new FormData();
 data.append("file", myFile, myFile.name);

 let xhr = new XMLHttpRequest();

 xhr.addEventListener("readystatechange", function () {
 if (this.readyState === 4) {
 let json_response = JSON.parse(this.responseText)
 setShapes(formatPrediction(json_response.predictions[0]))
 setReceiptData(json_response.predictions[0])
 }
 });

 xhr.open("POST", "https://api.mindee.net/products/expense_receipts/v2/predict");
 xhr.setRequestHeader("X-Inferuser-Token", "my-token-here");
 xhr.send(data);
 }

 return (
 &lt;div&gt;
 &lt;input type="file" onChange={e =&gt; parseReceipt(e)}/&gt;
 {
 image &amp;&amp; &lt;&gt;
 &lt;div style={{height: '500px', width: '400px'}}&gt;
 &lt;AnnotationViewer
 shapes={shapes}
 image={image}
 /&gt;
 &lt;/div&gt;
 {
 receiptData &amp;&amp;
 &lt;div style={{marginLeft: "50px"}}&gt;
 &lt;p&gt;Total amount : {receiptData.total.amount}&lt;/p&gt;
 &lt;p&gt;Date: {receiptData.date.iso}&lt;/p&gt;
 &lt;p&gt;Time: {receiptData.time.iso}&lt;/p&gt;
 &lt;p&gt;Category: {receiptData.category.value}&lt;/p&gt;
 &lt;/div&gt;
 }
 &lt;/&gt;
 }
 &lt;/div&gt;
 );
}</code></pre>

&nbsp;

Run the app, upload a receipt, you should get:

&nbsp;

<img alt="Reactjs mindee parsing receipt" src="https://mindee-public-website-staging.s3.amazonaws.com/blog/2020/09/04/hello_world_receipt_react_step.jpg" />

&nbsp;

We&#39;re almost done!

&nbsp;

<h2>Bind interractions between images boxes and corresponding receipt text features</h2>

&nbsp;

To make sure your user can validate very easily data extracted by the API, the ReactJS SDK allows you to add interactions with visual features on the images and the rest of your app.

&nbsp;

First, we&#39;ll create a function that change the isActive attribute inside a shape object, from it&#39;s name. The SDK will understand this attribute and will modify the shape layout (you can modify the active layout by changing colors and line width in the active attribute of a shape).

&nbsp;

<pre>
<code class="language-javascript">const highlightFeature = (featureName) =&gt; {
 let newShapes = []
 shapes.forEach(shape =&gt; {
 shape.isActive=(shape.featureName === featureName)
 newShapes.push(shape)
 }
 )
 setShapes(newShapes)
}</code></pre>

&nbsp;

We&#39;ll then modify a bit the rendered code, and bind on the onMouseOver event of the text features the highlight function.

&nbsp;

<pre>
<code class="language-javascript">return (
 &lt;div&gt;
 &lt;input type="file" onChange={e =&gt; parseReceipt(e)}/&gt;
 {
 image &amp;&amp; &lt;&gt;
 &lt;div style={{height: '500px', width: '400px'}}&gt;
 &lt;AnnotationViewer
 shapes={shapes}
 image={image}
 /&gt;
 &lt;/div&gt;
 {
 receiptData &amp;&amp;
 &lt;div style={{marginLeft: "50px"}}&gt;
 &lt;p onMouseOver={() =&gt; highlightFeature('total')}&gt;Total amount : {receiptData.total.amount}&lt;/p&gt;
 &lt;p onMouseOver={() =&gt; highlightFeature('date')}&gt;Date: {receiptData.date.iso}&lt;/p&gt;
 &lt;p onMouseOver={() =&gt; highlightFeature('time')}&gt;Time: {receiptData.time.iso}&lt;/p&gt;
 &lt;p&gt;Category: {receiptData.category.value}&lt;/p&gt;
 &lt;/div&gt;
 }
 &lt;/&gt;
 }
 &lt;/div&gt;
 );</code></pre>

&nbsp;

Final App.js

<pre>
<code class="language-javascript">import React, {useState} from 'react';
import './App.css';

import {
 AnnotationViewer,
 formatPrediction
} from "react-mindee-js";

const HelloWorld = () =&gt; {
 const [image, setImage] = useState(null)
 const [shapes, setShapes] = useState([])
 const [receiptData, setReceiptData] = useState(null)

 const parseReceipt = (e) =&gt; {
 let myFile = e.target.files[0]

 if (!myFile) {
 return
 }
 setImage(URL.createObjectURL(myFile))

 let data = new FormData();
 data.append("file", myFile, myFile.name);

 let xhr = new XMLHttpRequest();

 xhr.addEventListener("readystatechange", function () {
 if (this.readyState === 4) {
 let json_response = JSON.parse(this.responseText)
 setShapes(formatPrediction(json_response.predictions[0]))
 setReceiptData(json_response.predictions[0])
 }
 });

 xhr.open("POST", "https://api.mindee.net/products/expense_receipts/v2/predict");
 xhr.setRequestHeader("X-Inferuser-Token", "my-token-here");
 xhr.send(data);
 }

 const highlightFeature = (featureName) =&gt; {
 let newShapes = []
 shapes.forEach(shape =&gt; {
 shape.isActive=(shape.featureName === featureName)
 newShapes.push(shape)
 }
 )
 setShapes(newShapes)
 }

 return (
 &lt;div&gt;
 &lt;input type="file" onChange={e =&gt; parseReceipt(e)}/&gt;
 {
 image &amp;&amp; &lt;&gt;
 &lt;div style={{height: '500px', width: '400px'}}&gt;
 &lt;AnnotationViewer
 shapes={shapes}
 image={image}
 /&gt;
 &lt;/div&gt;
 {
 receiptData &amp;&amp;
 &lt;div style={{marginLeft: "50px"}}&gt;
 &lt;p onMouseOver={() =&gt; highlightFeature('total')}&gt;Total amount : {receiptData.total.amount}&lt;/p&gt;
 &lt;p onMouseOver={() =&gt; highlightFeature('date')}&gt;Date: {receiptData.date.iso}&lt;/p&gt;
 &lt;p onMouseOver={() =&gt; highlightFeature('time')}&gt;Time: {receiptData.time.iso}&lt;/p&gt;
 &lt;p&gt;Category: {receiptData.category.value}&lt;/p&gt;
 &lt;/div&gt;
 }
 &lt;/&gt;
 }
 &lt;/div&gt;
 );
}
export default HelloWorld;</code></pre>

&nbsp;

Running your app, you should see now that when you hover over a text line at the bottom,&nbsp;the color of the box highlighting the value on the receipt&nbsp;will change. You can modify the colours as you want to make the change more or less dramatic, as you please.&nbsp; All of the details for this is&nbsp;explained in the Github readme.

&nbsp;

<h2>Conclusion</h2>

&nbsp;

Using our ReactJS library makes identifying regions in the image easy.&nbsp; The Mindee Receipt Parsing API extracts the relevant details (and the location in the image). Then the react library highlights the locations with the AnnotationViewer.&nbsp;Combined, you have a powerful tool to extract data from images, and display them for verification to your users.

&nbsp;

&nbsp;

Tutorial on how to extract data from a receipt image through a reactjs user interface. You will learn how to parse the different data extracted by the Mindee API and how to create a beautiful user interface using Mindee ReactJS library.

reactjs

Parsing Receipts

<h1>Parsing Invoices with Mindee&rsquo;s API</h1>

&nbsp;

&nbsp;

Scanning and inputting data from invoices into accounting systems is a slow and tedious process.&nbsp; Using Mindee&rsquo;s invoice parsing API, all of the pertinent data can be automatically and accurately extracted from your invoice in seconds, allowing for fast and painless data entry.&nbsp;&nbsp;

&nbsp;

Interested in learning more? In this tutorial, we will walk through the steps to use Mindee&rsquo;s Invoice API.&nbsp;

&nbsp;

<h2>API Prerequisites</h2>

&nbsp;

<ol>
	<li style="list-style-type:decimal">You&rsquo;ll need Mindee account.<a href="http://platform.mindee.net/signup"> Sign up</a><a href="http://platform.mindee.net/signup">&nbsp;for free</a>.&nbsp; Then confirm your email to login.</li>
	<li style="list-style-type:decimal">An invoice.&nbsp; Use a recently received invoice, or do a <a href="https://www.google.com/search?q=invoice">Google Image search</a> for a invoice&nbsp;and download a few to test with.&nbsp;&nbsp;</li>
</ol>

&nbsp;

<h2>Setting up the API</h2>

&nbsp;

Log into you Mindee account and access your Invoices API environment by clicking the Invoices card :

&nbsp;

<h1><img alt="" src="https://mindee-public-website.s3.amazonaws.com/blog/2020/09/11/screenshot-2020-09-11-at-192210.png" style="height:558px; width:400px" /></h1>

To activate the API, click the &ldquo;Try for Free&rdquo; button to access the free tier. You&rsquo;ll land on the dashboard page - this page will show the usage of your account with the Invoice API - of course, it is empty now.&nbsp; On the left navigation, there are links to &ldquo;Documentation&rdquo;, &ldquo;Credentials&rdquo; and &ldquo;Live Interface&rdquo;.&nbsp; The docs tab has all of the technical details you&rsquo;ll need to build for the Invoce API endpoint, and the Live Interface is a cool interactive demo. Rather than try out the demo, we want to build with the API,&nbsp; so click on &ldquo;Credentials&rdquo; to create an API token.

&nbsp;

Add a new token. In this example, I&rsquo;ve named it &ldquo;Tutorial&rdquo;

<h1> 
<img alt="" src="https://mindee-public-website.s3.amazonaws.com/blog/2020/09/01/pasted-image-0-1.png" style="height:247px; width:500px" /></h1>

&nbsp;

Click &ldquo;Add New Key&rdquo; and you&rsquo;ll be able to see your API token.

Now, we are ready to make an API call.&nbsp; In this example, we&rsquo;ll be using cURL.

&nbsp;

<pre>
<code class="language-bash">curl -X POST \ https://api.mindee.net/products/invoices/v1/predict 
 -H 'X-Inferuser-Token: {apiToken}' 
 -H 'content-type: multipart/form-data; boundary=----WebKitFormBoundary7MA4YWxkTrZu0gW' 
 -F file=@/path/to/your/file.png</code></pre>

&nbsp;

&nbsp;

Simply replace ```{apiToken}``` with your new API token and ```/path/to/your/file/png``` with the path to your invoice - wither an image or a pdf.&nbsp;

&nbsp;

NOTE: You can also copy this code right from the documentation tab of the API with your API token inserted for you.

Here&rsquo;s the invoice I used - the roofer who fixed the gasket around our chimney this spring (I&rsquo;ve obfuscated my home address in the image):

&nbsp;

<h1><img alt="" src="https://mindee-public-website.s3.amazonaws.com/blog/2020/09/11/screenshot-2020-09-11-at-193913.png" style="height:383px; width:500px" /></h1>

&nbsp;

Pasting the cURL sample into my terminal, I hit enter and about a second later, I received a JSON response with the receipt details.&nbsp; The full JSON can be accessed here (link to JSON file).&nbsp; Since the response is quite verbose, we will walk through the various fields section by section.

&nbsp;

<h2>Extracted fields</h2>

&nbsp;

<h2>Summary &amp; Documents section</h2>

&nbsp;

The first two sections of the response contain information about the API call made:

<pre>
<code class="language-json">	"call": {
		"endpoint": { "name": "invoices", "version": "1.0" },
		"finished_at": "2020-09-11T18:39:58+00:00",
		"id": "3b4ac75a-f800-466b-bfc4-56c8f8e75134",
		"n_documents": 1,
		"n_inputs": 1,
		"processing_time": 1.047,
		"started_at": "2020-09-11T18:39:57+00:00"
	},
	"documents": [
		{
			"id": "5d72b0c2-16dd-481d-9e8b-031c5d1ab33a",
			"name": "Screenshot 2020-09-11 at 19.39.13.png"
		}
	],</code></pre>

&nbsp;

The call section tells us that we ran on the v1 of the invoice endpoint, uploading one document that is one page long.&nbsp; After a second, the file was processed, and the response transmitted back to me.&nbsp; The documents section gives the Mindee id for the file, and the filename (and you can tell I used a screenshot of the original invoice).

&nbsp;

<h2>Predictions:&nbsp;</h2>

&nbsp;

This is the fun part, where the API extraxcts pertinent details form teh invoice automatcilly, saving tedious manual data entry.&nbsp; Every invoice is different, and will have different fields. This invoice, being sent from the UNited States will not have some common European features like and IBAN or Tax Id.&nbsp;&nbsp;

Other peices are rpesent in this invoice, but not picked up by the API.&nbsp; We are planning multiple additional releases to this API in 2020 that will continue to improve the accuracy and precision of the results.&nbsp;&nbsp;

&nbsp;

<h3>Company Number</h3>

In this case, there is no company number in the invoice:

<pre>
<code class="language-json">"company_number": {
				"probability": 0,
				"segmentation": { "bounding_box": [] },
				"type": "N/A",
				"value": "N/A"
			},</code></pre>

<h2>&nbsp;</h2>

<h3>Due Date</h3>

My invoice does have a due date, but it was not extracted.&nbsp;

<pre>
<code class="language-json">			"due_date": {
				"iso": "N/A",
				"probability": 0,
				"raw": "N/A",
				"segmentation": { "bounding_box": [] }
			},</code></pre>

<h2>&nbsp;</h2>

<h3>Invoice Date</h3>

The work was done in late June of this year, and the API correctly extracts the invoice date, along with four [x,y] points boxing in the value:

<pre>
<code class="language-json">			"invoice_date": {
				"iso": "2020-06-29",
				"probability": 0.99,
				"segmentation": {
					"bounding_box": [
						[ 0.766, 0.392 ],
						[ 0.855, 0.392 ],
						[ 0.855, 0.416 ],
						[ 0.766, 0.416 ]
					]
				}
			},</code></pre>

<h3>&nbsp;</h3>

<h3>Invoice Number</h3>

The invoice is numbered 1277, which is extracted by the API, along with the four [x,y] corrdinates denoting the location in the image.

<pre>
<code class="language-json">			"invoice_number": {
				"probability": 0.99,
				"segmentation": {
					"bounding_box": [
						[ 0.766, 0.363 ],
						[ 0.807, 0.363 ],
						[ 0.807, 0.382 ],
						[ 0.766, 0.382 ]
					]
				},
				"value": "1277"
			},</code></pre>

<h3>&nbsp;</h3>

<h3>Locale</h3>

Based on the information on the invoice, the API is able to predict (with 82% confidence) that this bill originated in the United States, is in USD, and is in English.

<pre>
<code class="language-json">"locale": { "currency": "USD", "language": "en", "probability": 0.82 },</code></pre>

<h2>&nbsp;</h2>

<h3>Orientation</h3>

The invoice did not require rotation to be parsed.

<pre>
<code class="language-json">"orientation": { "degrees": 0, "probability": 0.99 },</code></pre>

<h2>&nbsp;</h2>

<h3>Payment Details</h3>

The invoice does not contain any payment details, so were not extracted.

<pre>
<code class="language-json">"payment_details": {
				"iban": "N/A",
				"probability": 0,
				"segmentation": { "bounding_box": [] }
			},</code></pre>

<h3>&nbsp;</h3>

<h3>Supplier</h3>

The API is only 50% confident in this result, but it does extract the name of the company (despite without any spaces in the name).&nbsp;&nbsp;

<pre>
<code class="language-json">			"supplier": {
				"probability": 0.5,
				"segmentation": {
					"bounding_box": [
						[ 0.446, 0.066 ],
						[ 0.561, 0.066 ],
						[ 0.561, 0.083 ],
						[ 0.446, 0.083 ]
					]
				},
				"value": "SILVERHAMMER"
			},</code></pre>

<h3>&nbsp;</h3>

<h3>Tax Id</h3>

Companies in the US generally do not list their Tax IDs on invoices, so this was not found.

<pre>
<code class="language-json">			"tax_id": {
				"probability": 0,
				"segmentation": { "bounding_box": [] },
				"value": "N/A"
			},</code></pre>

<h3>&nbsp;</h3>

<h3>Taxes</h3>

The API did not extract the texes, or the percentage tax rate (we have an update later this year t better extract US tax details.)

<pre>
<code class="language-json">"taxes": [],</code></pre>

<h2>&nbsp;</h2>

<h3>Total without Taxes</h3>

The API did not extract the taxes, but did get the cost before taxes with 99% confidence.&nbsp;&nbsp;

<pre>
<code class="language-json">	"total_excl": {
				"amount": 350,
				"probability": 0.99,
				"segmentation": {
					"bounding_box": [
						[ 0.874, 0.803 ],
						[ 0.931, 0.803 ],
						[ 0.931, 0.825 ],
						[ 0.874, 0.825 ]
					]
				}
			},</code></pre>

<h2>&nbsp;</h2>

<h3>Total with Taxes</h3>

The total with taxes was alos extracted accurately with 99% confidence, meaning that the tax value could easily be calculated with a simple difference.

<pre>
<code class="language-json">"total_incl": {
				"amount": 380.45,
				"probability": 0.99,
				"segmentation": {
					"bounding_box": [
						[ 0.842, 0.897 ],
						[ 0.933, 0.897 ],
						[ 0.933, 0.93 ],
						[ 0.842, 0.93 ]
					]
				}
			}</code></pre>

&nbsp;

<h2>Conclusion</h2>

Manual entry of invoices is a time consuming task. Adding the Mindee API to your system will simplify the extraction and make your accounts paybale processes run more accurately and smoothly.&nbsp; Do you have more questions? Click the chat button at the bottom right.

&nbsp;

<h1>&nbsp;</h1>

&nbsp;

&nbsp;

Inputting data from invoices into accounting systems is a slow and tedious process.  Using Mindee’s invoice parsing API, extract data from invoices automatically and accurately in seconds, allowing for fast and painless data entry.

<h1>Extract data from invoices with Mindee&#39;s API using Python&nbsp;</h1>

&nbsp;

In this tutorial, you will learn how to extract data from invoices (pdfs or images) using python. This can help you automatically process invoices in any accounting software.

&nbsp;

The final script can be used within any python middleware application or REST API serving your frontend interfaces.

&nbsp;

In the end of this tutorial, you will learn how to transform a pdf into an image file, and how to highlight features on it to make human validation easier.&nbsp;

&nbsp;

The final result looks like this:

&nbsp;

<img alt="extract data invoices python mindee api" src="https://mindee-public-website-staging.s3.amazonaws.com/blog/2020/09/12/invoice_python.gif" />

&nbsp;

&nbsp;

<h2>API Prerequisites</h2>

&nbsp;

<ol>
	<li>You&rsquo;ll need a free Mindee account. <a href="https://platform.mindee.net/signup">Sign up</a> and confirm your email to login.</li>
	<li>An invoice, look into your mailbox or <a href="https://www.google.com/search?q=sample+invoice">check out on google image</a></li>
</ol>

&nbsp;

<h2>Setup the project</h2>

&nbsp;

Create an empty directory wherever you want in your laptop, we&rsquo;ll call ours &ldquo;python_invoice&rsquo;, and create a &lsquo;main.py&rsquo; file in it.

&nbsp;

Open a terminal and go to the directory you just created. In this tutorial we&rsquo;ll use the python library &ldquo;requests&rdquo; to call the API. If it&rsquo;s not installed in your env, you can install by running:

&nbsp;

&nbsp;

<pre>
<code class="language-bash">pip install requests</code></pre>

&nbsp;

&nbsp;

Make sure you have an invoice somewhere on your laptop, and you know the path.

&nbsp;

That&rsquo;s it for the setup, let&rsquo;s call the API.

&nbsp;

&nbsp;

<h2>Call the Mindee Invoice API</h2>

&nbsp;

&nbsp;

Login onto the [Mindee dashboardt](https://platform.mindee.net/) and enter the&nbsp;Invoice API environment by clicking the following card.

&nbsp;

&nbsp;

<img alt="mindee invoice api card" src="https://lh3.googleusercontent.com/JwH29ZumYioDxoUEJ_9BcyZjIMRHq9b-xGf5FWrNS1ZyJ1HKZ66Ihwpxiyop3MB6tNchMr9Py4VbQk2cKQ80nk4Gy1Q0nOe0zs0avA101kBA1uagx9JmTQ2A0FIpC62MxNhi1rKz" />

&nbsp;

&nbsp;

If you don&rsquo;t have an API token yet for this API, go to the &ldquo;credentials&rdquo; section and create a new API Key.&nbsp;

&nbsp;

&nbsp;

<img alt="new api token" src="https://lh3.googleusercontent.com/CWRZZFfY2uVd5Lyx-wxxR_EjWcw6ZBryFhhpNMwP1DBO5LQEEUBfqos2sPjCE4BQ6AdxelSzaybQEl5EGGXDL-eqwr5qXA3dtgn8_TquDVfVp8xFMsjV_npDbTblZ-wXTgOkwoEy" />

&nbsp;

&nbsp;

Click on the documentation section in the navbar, and then click on the Python link on the sample code area. Copy the code.

&nbsp;

&nbsp;

&nbsp;<img alt="python sample code mindee api" src="https://lh4.googleusercontent.com/iVYp-B4GFz-EE7NqPsZ5NTFyOYCjyoKJwJ3RKSGiJNQL0cg0rzS2qW62N4LBvBDjKxYmWx2I7e7uS6RQKUP_-GtnqOc0wZXYygAiRwKLqTNxqEjK9WOj6BqbOgE0_ZyUJh1Qj7xC" />

&nbsp;

&nbsp;

Open the &ldquo;main.py&rdquo; file you created a few minutes ago, and paste your sample code.&nbsp; The path to your invoice will be placed in the variable &quot;filepath&quot; since we call it several times. Your main.py file&nbsp;should look&nbsp;like:

&nbsp;

&nbsp;

<pre>
<code class="language-python">import requests


filepath = "/path/to/your/file"
url = "https://api.mindee.net/products/invoices/v1/predict"

with open(filepath, "rb") as myfile:
 files = {"file": myfile}
 headers = {"X-Inferuser-Token": "my-token-here"}
 response = requests.post(url, files=files, headers=headers)
 print(response.text)
</code></pre>

&nbsp;

&nbsp;

Replace the &ldquo;path/to/my/file&rdquo; placeholder in the code with the path of the invoice pdf or image you want to parse. Replace the &ldquo;my-token-here&rdquo; with the API token you created previously in the platform.

&nbsp;

&nbsp;

You can get back to your console and run the script

&nbsp;

&nbsp;

<pre>
<code class="language-bash">python main.py</code></pre>

&nbsp;

&nbsp;

You should see a json object appearing in your console. Now let&rsquo;s parse this json to get the data we need.

&nbsp;

<h2>&nbsp;</h2>

<h2>Parse the result</h2>

&nbsp;

The API returns a list of different fields in the invoice (total amount, taxes, date &hellip;). You can find this list in the &ldquo;documentation&rdquo; section in Mindee platform.

&nbsp;

Now that we coded the API request, we&rsquo;ll use the json object and get a few extracted features.

&nbsp;

First, let&rsquo;s create a parsing function that returns the invoice total amount, the invoice date of issuance, and the taxes. We&rsquo;ll focus only on those three features for the tutorial but you can find more in the documentation. The features are inside the prediction object in the json response.

Note:&nbsp;As you can send a multi-page&nbsp;pdf to the API, the predictions object is an array, with a prediction object for each page.

&nbsp;

In our case, as we are sending a single page pdf, the predictions array contains only one object. As the taxes object returned is an array with different tax objects, we are just going to iterate over the different taxes and concatenate values in a string so we can easily display it.

&nbsp;

&nbsp;

<pre>
<code class="language-ruby">def get_features(json_response):
 parsed_data = {}
 prediction = json_response["predictions"][0]

 # get features values
 parsed_data["total_incl"] = prediction["total_incl"]["amount"]
 parsed_data["date"] = prediction["invoice_date"]["iso"]
 parsed_data["taxes"] = []
 for tax in prediction["taxes"]:
 parsed_data["taxes"].append(str(tax["amount"])+ " "+str(tax["rate"])+"%")

 return parsed_data</code></pre>

&nbsp;

&nbsp;

Now that our function is ready, we&rsquo;ll just print the result in the console and add a simple check to ensure the API call was well performed and we can access the result. You can replace the main.py script to this:

&nbsp;

&nbsp;

<pre>
<code class="language-ruby">import requests


def get_features(json_response):
 parsed_data = {}
 prediction = json_response["predictions"][0]

 # get features values
 parsed_data["total_incl"] = prediction["total_incl"]["amount"]
 parsed_data["date"] = prediction["invoice_date"]["iso"]
 parsed_data["taxes"] = []
 for tax in prediction["taxes"]:
 parsed_data["taxes"].append(str(tax["amount"])+ " "+str(tax["rate"])+"%")

 return parsed_data


filepath = "/path/to/your/file"
url = "https://api.mindee.net/products/invoices/v1/predict"

with open(filepath, "rb") as myfile:
 files = {"file": myfile}
 headers = {"X-Inferuser-Token": "my-token-here"}
 response = requests.post(url, files=files, headers=headers)

 if response.status_code != 200:
 print("Request error")
 else:
 json_response = response.json()
 features = get_features(json_response)
 print("Date:", features["date"])
 print("Taxes: ", ' - '.join(features["taxes"]))
 print("Total amount including taxes:", features["total_incl"])</code></pre>

 
&nbsp;

Run the script and check out the results on your image.

&nbsp;&nbsp;

Now we are going to add a piece of code that will highlight the features extracted on the pdf. It can help you or your users to very quickly validate the data extraction.

&nbsp;

&nbsp;

<h2>Convert pdf into an image</h2>

&nbsp;

If you&rsquo;re using a pdf, we need first to convert the pdf into an image, as it&rsquo;s more convenient to use for highlighting the features on it. If your invoice is an image, you can skip this part.

&nbsp;

To convert the pdf into a jpeg we are going to use the <a href="https://github.com/Belval/pdf2image">pdf2image</a>&nbsp;python library; First install it in your environment:

&nbsp;

<pre>
<code class="language-bash">pip install pdf2image</code></pre>

&nbsp;

Important note: Dealing with pdfs is not easy and this library might not work directly because it sometimes requires some extra dependencies. Check out the github to make sure you installed it correctly.

&nbsp;

Once installed, we are going to create an image out of our pdf:

&nbsp;

&nbsp;

<pre>
<code class="language-python">from pdf2image import convert_from_path, convert_from_bytes
images = convert_from_path(filepath)</code></pre>

&nbsp;

&nbsp;

The images object is an array of Pillow images. Pillow is an image library for python. As we want to use openCV for highlighting the features on the image, we need to convert the pillow image into an OpenCV image. OpenCV is based on the numpy library, and to convert it we are going to use numpy directly.

&nbsp;

If you don&rsquo;t have numpy installed on your environment:

&nbsp;

&nbsp;

<pre>
<code class="language-bash">pip install numpy</code></pre>

&nbsp;

&nbsp;

Now let&rsquo;s convert the Pillow image into an opencv format using numpy:

&nbsp;

&nbsp;

<pre>
<code class="language-ruby">import numpy as np
from pdf2image import convert_from_path
 
def pdf_to_cv_image(filepath, page=0):
 image = convert_from_path(filepath)[page]
 return np.asarray(image)</code></pre>

 
 
&nbsp;

<h2>Highlight features on the image</h2>

&nbsp;

Let&rsquo;s try to highlight the features as if someone did it with a pen.

&nbsp;

First, you&rsquo;ll need to install the computer vision python library OpenCV if you don&rsquo;t have it already installed in your env. To do so, run:

&nbsp;

&nbsp;

<pre>
<code class="language-bash">pip install opencv-python</code></pre>

&nbsp;

&nbsp;

We need to change a bit our get_features function to return coordinates for each feature, as we&#39;re using this to know where to draw the rectangles.

&nbsp;

&nbsp;

<pre>
<code class="language-ruby">def get_features(json_response):
 parsed_data = {}
 coordinates = []

 prediction = json_response["predictions"][0]

 # get features values
 parsed_data["total_incl"] = prediction["total_incl"]["amount"]
 parsed_data["date"] = prediction["invoice_date"]["iso"]
 parsed_data["taxes"] = []
 for tax in prediction["taxes"]:
 parsed_data["taxes"].append(str(tax["amount"])+ " "+str(tax["rate"])+"%")

 # get features bounding boxes
 coordinates.append(prediction["total_incl"]["segmentation"]["bounding_box"])
 coordinates.append(prediction["invoice_date"]["segmentation"]["bounding_box"])

 for tax in prediction["taxes"]:
 coordinates.append(tax["segmentation"]["bounding_box"])

 return parsed_data, coordinates</code></pre>

&nbsp;

&nbsp;

Once it&rsquo;s done, we are going to create our highlighter function in different steps, as follows:

&nbsp;

<ol>
	<li>Check if the file is a pdf or an image and load it as an openCV image in both scenarios</li>
	<li>Create a mask image</li>
	<li>Loop on each feature coordinates and draw the feature rectangle on our mask</li>
	<li>Overlay the mask and original image with alpha</li>
	<li>Display image to the user</li>
</ol>

&nbsp;

Note: each coordinate returned by the API is relative (in % of the image). You&#39;ll see there is a relative to absolute conversion in the code.

&nbsp;

Here is the code step by step:

&nbsp;

<pre>
<code class="language-ruby">def highlight_features(img_path, coordinates):
 # step 1: Check if the file is a pdf or an image and load it as an openCV image in both scenarios
 if img_path.endswith(".pdf"):
 cv_image = pdf_to_cv_image(img_path)
 else:
 cv_image = cv2.imread(img_path)

 # step 2: create mask image
 overlay = cv_image.copy()
 h, w = cv_image.shape[:2]

 # step 3: Loop on each feature coordinates and draw the feature rectangle on our mask
 for coord in coordinates:
 if len(coord):
 pt1 = (int(w*coord[0][0]), int(h*coord[0][1]))
 pt2 = (int(w*coord[2][0]), int(h*coord[2][1]))
 cv2.rectangle(overlay, pt1, pt2, (70, 230, 244), cv2.FILLED)

 # step 4: Overlay the mask and original image with alpha
 final_image = cv2.addWeighted(overlay, 0.5, cv_image, 0.5, 0)

 # step 5: Display image to the user
 cv2.imshow("highlghted_image", cv2.resize(final_image, (600, int(600*h/w))))
 cv2.waitKey(0)</code></pre>

 
&nbsp;

&nbsp;

Finally, we just have to modify a bit the code for executing the highlighting function and printing the results. Here is what the final code:

&nbsp;

&nbsp;

<pre>
<code class="language-ruby">import requests
import cv2
import numpy as np
from pdf2image import convert_from_path


def pdf_to_cv_image(filepath, page=0):
 image = convert_from_path(filepath)[page]
 return np.asarray(image)


def get_features(json_response):
 parsed_data = {}
 coordinates = []

 prediction = json_response["predictions"][0]

 # get features values
 parsed_data["total_incl"] = prediction["total_incl"]["amount"]
 parsed_data["date"] = prediction["invoice_date"]["iso"]
 parsed_data["taxes"] = []
 for tax in prediction["taxes"]:
 parsed_data["taxes"].append(str(tax["amount"])+ " "+str(tax["rate"])+"%")

 # get features bounding boxes
 coordinates.append(prediction["total_incl"]["segmentation"]["bounding_box"])
 coordinates.append(prediction["invoice_date"]["segmentation"]["bounding_box"])

 for tax in prediction["taxes"]:
 coordinates.append(tax["segmentation"]["bounding_box"])

 return parsed_data, coordinates


def highlight_features(img_path, coordinates):
 # step 1: Check if the file is a pdf or an image and load it as an openCV image in both scenarios
 if img_path.endswith(".pdf"):
 cv_image = pdf_to_cv_image(img_path)
 else:
 cv_image = cv2.imread(img_path)

 # step 2: create mask image
 overlay = cv_image.copy()
 h, w = cv_image.shape[:2]

 # step 3: Loop on each feature coordinates and draw the feature rectangle on our mask
 for coord in coordinates:
 if len(coord):
 pt1 = (int(w*coord[0][0]), int(h*coord[0][1]))
 pt2 = (int(w*coord[2][0]), int(h*coord[2][1]))
 cv2.rectangle(overlay, pt1, pt2, (70, 230, 244), cv2.FILLED)

 # step 4: Overlay the mask and original image with alpha
 final_image = cv2.addWeighted(overlay, 0.5, cv_image, 0.5, 0)

 # step 5: Display image to the user
 cv2.imshow("highlghted_image", cv2.resize(final_image, (600, int(600*h/w))))
 cv2.waitKey(0)


filepath = "/path/to/your/file"
url = "https://api.mindee.net/products/invoices/v1/predict"

with open(filepath, "rb") as myfile:
 files = {"file": myfile}
 headers = {"X-Inferuser-Token": "my-token-here"}
 response = requests.post(url, files=files, headers=headers)

 if response.status_code != 200:
 print("Request error")
 else:
 json_response = response.json()
 features, coords = get_features(json_response)
 print("Date:", features["date"])
 print("Taxes: ", ' - '.join(features["taxes"]))
 print("Total amount including taxes:", features["total_incl"])
 highlight_features(filepath, coords)
</code></pre>

 
&nbsp;

And the final result!

 
<img alt="extract data from invoices mindee python" src="https://lh6.googleusercontent.com/ODt5ys1bxWF8UqFH6aCNxEi09pE6LQD4qZXnG-xxN0QAxaPr-VHPTXZuFRS75DwW60zv5XqZrjZ4TRK9QJ7i358OH9HHGy5aIqiyMscoqEWgvSfdjGBpXQV-QGOe4eruJdcKFKou" />

&nbsp;

<h2>Conclusion</h2>

&nbsp;

In just over a few seconds, an invoice was parsed and the features were highlighted.&nbsp; 
&nbsp;

If you wanted to use this kind of script to display results to your users, i&#39;d advise you to do the highlighting in the front-end application, as sending images back from your middleware is not the best option because of payload sizes. The other solution would be to store the final image using cv2.imwrite(...) but it would make your client download the result.&nbsp;

&nbsp;

If you have questions, please reach out to us in the chat widget in the bottom right.

Tutorial on how to extract data from invoices (pdfs or images) using python. This can help you automatically parse invoices in any web applications.

Parsing Invoices

<h1>Parsing international passports with Mindee&#39;s API</h1>

&nbsp;

Many onboarding processes in mobile or web apps require to extract some data from ID documents. In this tutorial, you will learn how to automatically extract data from passports to offer to your users the best onboarding experience.

&nbsp;

We will walk through the steps to use Mindee&rsquo;s Internation Passport&nbsp;Parsing API.&nbsp; Let&rsquo;s get started!&nbsp;

&nbsp;

<h2>API Prerequisites</h2>

<ol>
	<li>You&rsquo;ll need a free Mindee account. <a href="https://platform.mindee.net/signup" target="_blank">Sign up</a>&nbsp;and confirm your email to login.</li>
	<li>A picture&nbsp;of a passport.&nbsp; You can use yours&nbsp;safely as data protection is one of our priority and we won&#39;t send your images to any third party application. You can also download a fake one <a href="https://mindee-public-website.s3.amazonaws.com/blog/2020/09/03/fake_passport.jpg">here</a>.</li>
</ol>

&nbsp;

&nbsp;

<img alt="" src="https://mindee-public-website-dev.s3.amazonaws.com/blog/2020/08/07/passport_pic.jpg" />

&nbsp;

&nbsp;

<h2>Setting up the API</h2>

&nbsp;

Log into your Mindee account and access your passport API environment by clicking the International&nbsp;Passport&nbsp;card:

&nbsp;

&nbsp;

<img alt="passport api mindee" src="https://mindee-public-website-dev.s3.amazonaws.com/blog/2020/08/07/passport_article.jpg" />

&nbsp;

To activate the API, click the &ldquo;Try for Free&rdquo; button. This will give you access the API for free 50 times a month.&nbsp; There are four sections on the API landing page, noted in the left navigation. You are currently on the dashboard, and there&nbsp;are additional links to &ldquo;Documentation&rdquo;, &ldquo;Credentials&rdquo; and &ldquo;Live Interface&rdquo;.&nbsp; The docs tab has all of the technical details you&rsquo;ll need to build for the passports API endpoint, and the Live Interface is a cool interactive demo.

We&#39;ll start by clicking &quot;credentials&quot; and creating a new API token, in this case, named Tutorial:

&nbsp;

<img src="https://mindee-public-website.s3.amazonaws.com/blog/2020/09/01/pasted-image-0-1.png" style="height:198px; width:400px" />

&nbsp;

Click &ldquo;Add New Key&rdquo; and you&rsquo;ll be able to see your API token,.

&nbsp;

If you move back to the documentation tab, you can pick your API token, and the language, and create an API call.&nbsp; Here is the API call in cURL:

<pre>
<code class="language-bash">curl -X POST \ https://api.mindee.net/products/passport/v1/predict \ 
-H 'X-Inferuser-Token: {apiToken}’ \
-H 'content-type: multipart/form-data; boundary=----WebKitFormBoundary7MA4YWxkTrZu0gW' \
-F file=@/path/to/your/file.png</code></pre>

&nbsp;

Simply replace {apiToken}&nbsp;with your new API token and /path/to/your/file/png with the path to your receipt.&nbsp;

 
That&#39;s it! We&#39;re all set to run this command in the terminal, and see the data that Mindee can extract.

In this tutirla, I&#39;ll use a sample [Cyprus passport](https://commons.wikimedia.org/wiki/File:Cyprus_passport_data_page.jpg) from Wikimedia.

<img alt="Cyprus Passport used in this example" src="https://mindee-public-website.s3.amazonaws.com/blog/2020/09/04/1024px-cyprus_passport_data_page.jpg" style="height:334px; width:514px" />

<h2>&nbsp;</h2>

<h2>Extracted fields</h2>

&nbsp;

The JSON response can be a bit verbose, so we will break it into sections to describe what we are seeing.&nbsp; The initial sections describe the API and the API response:

<pre>
<code class="language-json">"call": {
		"endpoint": {
			"name": "passport",
			"version": "1.0"
		},
		"finished_at": "2020-09-04T20:57:51+00:00",
		"id": "7d81f5fe-fce1-4b63-b3de-cc5948c39b55",
		"n_documents": 1,
		"n_inputs": 1,
		"processing_time": 1.988,
		"started_at": "2020-09-04T20:57:49+00:00"
	},
	"documents": [
		{
			"id": "57aded44-a216-45be-859c-f949cc5b011d",
			"name": "1024px-Cyprus_passport_data_page.jpg"
		}
	],</code></pre>

Here, we can see that passport API was called, and it processed one document in about 2 seconds.&nbsp; What we are really interested in seeing is the results, and these can be found in the predictions section of the JSON file:

<h2>&nbsp;</h2>

<h2>Predictions</h2>

&nbsp;

The API response is in alphabetical order of items extracted from the document:

&nbsp;

<h3>Birth Date</h3>

<pre>
<code class="language-json">"birth_date": {
				"probability": 1.0,
				"segmentation": {
					"bounding_box": [
						[.053,	0.912],
						[0.955,0.912],
						[0.955,0.969],
						[0.053,0.969]
					]
				},
				"value": "1970-01-01"
			},</code></pre>

The API matched the brithdate in the MRX fields (more on that in a minute), so is 100% confident that the birthdate is 1 January, 1970. The bounding box gives four (x,y) points in the image where the date of birth can be found.

&nbsp;

<h3>Country of issuance</h3>

The API then attempts to parse the country that issued the passport.&nbsp;

<pre>
<code class="language-json">"country": {
				"probability": 0.99,
				"segmentation": {
					"bounding_box": [
						[0.479,0.199],
						[0.522,0.199],
						[0.522,0.234],
						[0.479,0.234]
					]
				},
				"value": "CYP"
			},</code></pre>

&nbsp;

The API is 99% certain that Cyprus is the issuing country. This is really incredible, as the API team tells me that there are no Cyprus passports in the data training set. Again, 4 (x,y) coordinates in the image point out where CYP is used.

<h3>&nbsp;</h3>

<h3>Expiration Date</h3>

<pre>
<code class="language-json">"expiry_date": {
				"probability": 1.0,
				"segmentation": {
					"bounding_box": [
						[0.053,0.912],
						[0.955,0.912],
						[0.955,0.969],
						[0.053,0.969]
					]
				},
				"value": "2020-12-01"
			},</code></pre>

Many countries do not allow entry with under six months of validity left on the passport.&nbsp; This woman should probably start the renewal process,&nbsp; as the API has identified that this passport will expire on 1 December, 2020.

<h3>&nbsp;</h3>

<h3>Gender</h3>

<pre>
<code class="language-json">			"gender": {
				"probability": 0.1,
				"segmentation": {
					"bounding_box": [
						[0.053,0.912],
						[0.955,0.912],
						[0.955,0.969],
						[0.053,0.969]
					]
				},
				"value": "F"
			},</code></pre>

The API&#39;s confidence is not high on this one, just 10%, but it does correctly identify the gender as female.

<h3>&nbsp;</h3>

<h3>Given Name</h3>

<pre>
<code class="language-json">			"given_names": [
				{
					"probability": 0.99,
					"segmentation": {
						"bounding_box": [
							[0.048,0.844],
							[0.952,0.844],
							[0.952,0.895],
							[0.048,0.895]
						]
					},
					"value": "AFRODITI"
				}
			],</code></pre>

The API is really certain (99% confidence) that the given name on the passport is Afroditi, meaning that our passport holder was named after the Greek goddess of beauty and love.

<h3>&nbsp;</h3>

<h3>ID Number</h3>

<pre>
<code class="language-json">			"id_number": {
				"probability": 1.0,
				"segmentation": {
					"bounding_box": [
						[0.673,0.197],
						[0.774,0.197],
						[0.774,0.233],
						[0.673,0.233]
					]
				},
				"value": "K00000413"
			},</code></pre>

While we are not numbers, in many ways, we are identified by numbers. In this case, the passport ID number, which must be used when booking flights, checking into hotels, etc.&nbsp; The API has found the location (marked by the 4 (x,y) points in the bounding box, for the Passport number, which is K00000413.

&nbsp;

<h3>Issuance Date</h3>

Contining the alphabetical trip through the passport fields, we next come to the date the passport was issued:

&nbsp;

<pre>
<code class="language-json">			"issuance_date": {
				"probability": 0.18,
				"segmentation": {
					"bounding_box": [
						[0.353,0.586],
						[0.426,0.586],
						[0.426,0.62],
						[0.353,0.62]
					]
				},
				"value": "2014-01-21"
			},</code></pre>

In this case, the API missed the issuance date.&nbsp; However, while our API does miss in it&#39;s predictions, in this case, the issue is due to the fact that the passport is a sample (we&#39;ll have a blog post on this soon).&nbsp; The issuance date is missing from the MRZ zone of the sample&nbsp;passport, so the API is unable to accurately measure the issuance date.

&nbsp;

<h3>Surname</h3>

<pre>
<code class="language-json">"surname": {
				"probability": 0.98,
				"segmentation": {
					"bounding_box": [
						[0.33,0.272],
						[0.444,0.272],
						[0.444,0.309],
						[0.33,0.309]
					]
				},
				"value": "ANONYMOU"
			}</code></pre>

Our fake passport holder has a surname of Anonymou, which is correctly identified by the API, along with bounding boxes.

&nbsp;

<h3>MRZ1 and MRZ2</h3>

The last two regions identified by the API (which I pulled *slightly* out of alphabetical order for readability) are the 2 Machine Readable Zone (MRZ) lines:

<pre>
<code class="language-json">"mrz1": {
				"probability": 0.99,
				"segmentation": {
					"bounding_box": [
						[0.048,0.844],
						[0.952,0.844],
						[0.952,0.895],
						[0.048,0.895]
					]
				},
				"value": "P&lt;CYPANONYMOU&lt;&lt;AFRODITI&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;"
			},
"mrz2": {
				"probability": 0.1,
				"segmentation": {
					"bounding_box": [
						[0.053,0.912],
						[0.955,0.912],
						[0.955,0.969],
						[0.053,0.969]
					]
				},
				"value": "K000004134CYP7001017F2012010&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;4"
			},</code></pre>

The MRZ zones (if you cannot tell from the JSON above) are the two lines of machine readable text at the bottom of every passport page.&nbsp; MRZ1 is line one, and MRZ2 is line 2.&nbsp; Most of the identifying information can be pulled from this area.&nbsp; The confidence interval for line one is high, as the fake passport has this area filled out correctly.&nbsp; However, the confidence level is low, as the algorithm expected more characters other than &quot;&lt;&quot; to supply the issuance date.

&nbsp;

<h2>Conclusion</h2>

&nbsp;

And there you have it! Mindee&#39;s passport API parsed a passport in under 2 seconds, matching nearly all of the sections of the passport quickly and accurately. Give it&nbsp;try with our free tier, and let us know in the chat how it works for you!

Many onboarding processes in mobile or web apps require to extract some data from ID documents. In this tutorial, you will learn how to automatically extract data from passports to offer to your users the best onboarding experience.

Code	Possible reasons
401 (Unauthorized)	Bad token: you didn’t provide the authentication header or the token is wrong.
400 (Bad request)	Wrong file type: file types allowed: png, jpg, webp, pdf Missing input file: see How to manage input files documentation Too many pages in pdf: APIs have a maximum limit of pages allowed for pdf files File is too big: APIs have a maximum file size allowed
404 (Not found)	Wrong version: wrong version in base URL
429 (Too many requests)	Plan limit reached: you have reached your maximum number of requests. Too high frequency: number of requests over time is limited. See Authentication documentation