Recently I participated in very interesting workshop on Technical University in GDN. Workshop was lead by Tomasz Kocejko from faculty: Medical Telematics. This workshop presented possibilites of using Machine Learning models on mobile devices. More info:
https://aibay.ai/wp-content/uploads/2020/11/kursy_Inferencja_Android_DIH.pdf
This is only an example what could be achieved with Transfer Learning. The goal is:
- creation of mobile app in language Dart using framework Flutter (new Google language for creating mobile apps), which will acquire data from camera or gallery and on this basis classify image. Images will be classified using binary perceptron, which will classify data into 2 sets: cats and dogs and when classification is uncertain unknown. This will be achieved by transfer learning from ready-to-use Mobilenet model.
For best results use computer with following configuration:
- Win 10
- 32GB RAM
- 1TB SSD Disk
- AMD Ryzen 7, 8 cores
First of all install:
- Android Studio
- Flutter
- Anaconda
- Visual Studio Code
Open Anaconda terminal and type:
conda create --name NameOfYourEnv
Then install following packages :
conda install -y _libgcc_mutex conda install -y _openmp_mutex conda install -y _r-mutex conda install -y _tflow_select conda install -y absl-py conda install -y aiohttp conda install -y alabaster conda install -y anyio conda install -y argh conda install -y argon2-cffi conda install -y astor conda install -y astroid conda install -y async-timeout conda install -y async_generator conda install -y atomicwrites conda install -y attrs conda install -y autopep8 conda install -y babel conda install -y backcall conda install -y backports conda install -y backports.shutil_get_terminal_size conda install -y binutils_impl_linux-64 conda install -y binutils_linux-64 conda install -y blas conda install -y bleach conda install -y blinker conda install -y bokeh conda install -y brotlipy conda install -y bwidget conda install -y bzip2 conda install -y c-ares conda install -y ca-certificates conda install -y cachetools conda install -y cairo conda install -y certifi conda install -y cffi conda install -y chardet conda install -y click conda install -y cloudpickle conda install -y conda conda install -y conda-package-handling conda install -y cryptography conda install -y curl conda install -y cycler conda install -y cyrus-sasl conda install -y cython conda install -y cytoolz conda install -y dask conda install -y dask-core conda install -y dbus conda install -y decorator conda install -y defusedxml conda install -y diff-match-patch conda install -y distributed conda install -y docutils conda install -y entrypoints conda install -y expat conda install -y fastcache conda install -y ffmpeg conda install -y flake8 conda install -y font-ttf-dejavu-sans-mono conda install -y font-ttf-inconsolata conda install -y font-ttf-source-code-pro conda install -y font-ttf-ubuntu conda install -y fontconfig conda install -y fonts-anaconda conda install -y freeglut conda install -y freetype conda install -y fribidi conda install -y future conda install -y gast conda install -y gcc-dbg_linux-64 conda install -y gcc_impl_linux-64 conda install -y gcc_linux-64 conda install -y gettext conda install -y gfortran-dbg_linux-64 conda install -y gfortran_impl_linux-64 conda install -y gfortran_linux-64 conda install -y glib conda install -y gmp conda install -y gmpy2 conda install -y google-auth conda install -y google-auth-oauthlib conda install -y google-pasta conda install -y graphite2 conda install -y grpcio conda install -y gsl conda install -y gst-plugins-base conda install -y gst-plugins-good conda install -y gstreamer conda install -y gxx_impl_linux-64 conda install -y gxx_linux-64 conda install -y h5py conda install -y harfbuzz conda install -y hdf5 conda install -y heapdict conda install -y html5lib conda install -y icu conda install -y idna conda install -y imageio conda install -y imagesize conda install -y importlib-metadata conda install -y importlib_metadata conda install -y intel-openmp conda install -y intervaltree conda install -y ipykernel conda install -y ipyparallel conda install -y ipython conda install -y ipython_genutils conda install -y ipywidgets conda install -y isort conda install -y jasper conda install -y jedi conda install -y jeepney conda install -y jinja2 conda install -y joblib conda install -y jpeg conda install -y json5 conda install -y jsonschema conda install -y jupyter conda install -y jupyter_client conda install -y jupyter_console conda install -y jupyter_core conda install -y jupyter_server conda install -y jupyterlab conda install -y jupyterlab_pygments conda install -y jupyterlab_server conda install -y keras conda install -y keras-applications conda install -y keras-base conda install -y keras-preprocessing conda install -y keyring conda install -y kiwisolver conda install -y krb5 conda install -y lazy-object-proxy conda install -y lcms2 conda install -y ld_impl_linux-64 conda install -y libcurl conda install -y libdb conda install -y libedit conda install -y libffi conda install -y libgcc-ng conda install -y libgfortran-ng conda install -y libglu conda install -y libgomp conda install -y libiconv conda install -y libntlm conda install -y libopencv conda install -y libopus conda install -y libpng conda install -y libprotobuf conda install -y libsodium conda install -y libspatialindex conda install -y libssh2 conda install -y libstdcxx-ng conda install -y libtiff conda install -y libuuid conda install -y libvpx conda install -y libxcb conda install -y libxml2 conda install -y llvmlite conda install -y locket conda install -y lz4-c conda install -y make conda install -y markdown conda install -y markdown2 conda install -y markupsafe conda install -y matplotlib conda install -y matplotlib-base conda install -y mccabe conda install -y mistune conda install -y mkl conda install -y mkl-service conda install -y mkl_fft conda install -y mkl_random conda install -y mpc conda install -y mpfr conda install -y mpmath conda install -y msgpack-python conda install -y multidict conda install -y nbclassic conda install -y nbclient conda install -y nbconvert conda install -y nbformat conda install -y ncurses conda install -y nest-asyncio conda install -y networkx conda install -y nose conda install -y notebook conda install -y numba conda install -y numexpr conda install -y numpy conda install -y numpy-base conda install -y numpydoc conda install -y oauthlib conda install -y olefile conda install -y opencv conda install -y openjdk conda install -y openssl conda install -y opt_einsum conda install -y packaging conda install -y pandas conda install -y pandoc conda install -y pandocfilters conda install -y pango conda install -y parso conda install -y partd conda install -y path conda install -y path.py conda install -y pathtools conda install -y patsy conda install -y pcre conda install -y pexpect conda install -y pickleshare conda install -y pillow conda install -y pip conda install -y pixman conda install -y pluggy conda install -y portaudio conda install -y prometheus_client conda install -y prompt-toolkit conda install -y prompt_toolkit conda install -y protobuf conda install -y psutil conda install -y ptyprocess conda install -y py-opencv conda install -y pyasn1 conda install -y pyasn1-modules conda install -y pyaudio conda install -y pycodestyle conda install -y pycosat conda install -y pycparser conda install -y pydocstyle conda install -y pyflakes conda install -y pygments conda install -y pyjwt conda install -y pylint conda install -y pyopenssl conda install -y pyparsing conda install -y pyqt conda install -y pyrsistent conda install -y pysocks conda install -y python=3.7.0 conda install -y python-dateutil conda install -y python-jsonrpc-server conda install -y python-language-server conda install -y python_abi conda install -y pytz conda install -y pywavelets conda install -y pyxdg conda install -y pyyaml conda install -y pyzmq conda install -y qdarkstyle conda install -y qt conda install -y qtawesome conda install -y qtconsole conda install -y qtpy conda install -y r-abind conda install -y r-askpass conda install -y r-assertthat conda install -y r-backports conda install -y r-base conda install -y r-base64enc conda install -y r-bh conda install -y r-bit conda install -y r-bit64 conda install -y r-bitops conda install -y r-blob conda install -y r-boot conda install -y r-broom conda install -y r-callr conda install -y r-car conda install -y r-cardata conda install -y r-catools conda install -y r-cellranger conda install -y r-cli conda install -y r-clipr conda install -y r-colorspace conda install -y r-config conda install -y r-corrplot conda install -y r-cowplot conda install -y r-crayon conda install -y r-curl conda install -y r-data.table conda install -y r-dbi conda install -y r-dbplyr conda install -y r-desc conda install -y r-digest conda install -y r-dplyr conda install -y r-ellipsis conda install -y r-evaluate conda install -y r-fansi conda install -y r-farver conda install -y r-forcats conda install -y r-foreign conda install -y r-forge conda install -y r-generics conda install -y r-ggplot2 conda install -y r-ggpubr conda install -y r-ggrepel conda install -y r-ggsci conda install -y r-ggsignif conda install -y r-glue conda install -y r-gridextra conda install -y r-gtable conda install -y r-haven conda install -y r-highr conda install -y r-hms conda install -y r-htmltools conda install -y r-htmlwidgets conda install -y r-httpuv conda install -y r-httr conda install -y r-isoband conda install -y r-jsonlite conda install -y r-knitr conda install -y r-labeling conda install -y r-later conda install -y r-lattice conda install -y r-lifecycle conda install -y r-lme4 conda install -y r-magrittr conda install -y r-maptools conda install -y r-markdown conda install -y r-mass conda install -y r-matrix conda install -y r-matrixmodels conda install -y r-mgcv conda install -y r-mime conda install -y r-miniui conda install -y r-minqa conda install -y r-mongolite conda install -y r-munsell conda install -y r-nlme conda install -y r-nloptr conda install -y r-nnet conda install -y r-odbc conda install -y r-openssl conda install -y r-openxlsx conda install -y r-packrat conda install -y r-pbkrtest conda install -y r-pillar conda install -y r-pkgbuild conda install -y r-pkgconfig conda install -y r-pkgload conda install -y r-pki conda install -y r-plyr conda install -y r-polynom conda install -y r-praise conda install -y r-prettyunits conda install -y r-processx conda install -y r-profvis conda install -y r-progress conda install -y r-promises conda install -y r-ps conda install -y r-purrr conda install -y r-quantreg conda install -y r-r2d3 conda install -y r-r6 conda install -y r-rappdirs conda install -y r-rcolorbrewer conda install -y r-rcpp conda install -y r-rcppeigen conda install -y r-rcurl conda install -y r-readr conda install -y r-readxl conda install -y r-rematch conda install -y r-reshape2 conda install -y r-rio conda install -y r-rjava conda install -y r-rjdbc conda install -y r-rjsonio conda install -y r-rlang conda install -y r-rmarkdown conda install -y r-rprojroot conda install -y r-rsconnect conda install -y r-rstatix conda install -y r-rstudioapi conda install -y r-scales conda install -y r-shiny conda install -y r-sourcetools conda install -y r-sp conda install -y r-sparklyr conda install -y r-sparsem conda install -y r-statmod conda install -y r-stringi conda install -y r-stringr conda install -y r-sys conda install -y r-testthat conda install -y r-tibble conda install -y r-tidyr conda install -y r-tidyselect conda install -y r-tinytex conda install -y r-utf8 conda install -y r-vctrs conda install -y r-viridislite conda install -y r-withr conda install -y r-xfun conda install -y r-xml2 conda install -y r-xtable conda install -y r-yaml conda install -y r-zeallot conda install -y r-zip conda install -y readline conda install -y requests conda install -y requests-oauthlib conda install -y rope conda install -y rsa conda install -y rstudio conda install -y rtree conda install -y ruamel conda install -y ruamel.yaml conda install -y ruamel.yaml.clib conda install -y ruamel_yaml conda install -y scikit-image conda install -y scikit-learn conda install -y scipy conda install -y seaborn conda install -y secretstorage conda install -y send2trash conda install -y setuptools conda install -y simplegeneric conda install -y sip conda install -y six conda install -y sniffio conda install -y snowballstemmer conda install -y sortedcontainers conda install -y sphinx conda install -y sphinxcontrib-applehelp conda install -y sphinxcontrib-devhelp conda install -y sphinxcontrib-htmlhelp conda install -y sphinxcontrib-jsmath conda install -y sphinxcontrib-qthelp conda install -y sphinxcontrib-serializinghtml conda install -y spyder conda install -y spyder-kernels conda install -y sqlite conda install -y statsmodels conda install -y sympy conda install -y tbb conda install -y tblib conda install -y tensorboard conda install -y tensorboard-plugin-wit conda install -y tensorflow conda install -y tensorflow-base conda install -y tensorflow-estimator=2.1.0 conda install -y termcolor conda install -y terminado conda install -y testpath conda install -y tk conda install -y tktable conda install -y toml conda install -y toolz conda install -y tornado conda install -y tqdm conda install -y traitlets conda install -y typed-ast conda install -y typing-extensions conda install -y typing_extensions conda install -y ujson conda install -y unixodbc conda install -y urllib3 conda install -y watchdog conda install -y wcwidth conda install -y webencodings conda install -y werkzeug conda install -y wfdb conda install -y wheel conda install -y widgetsnbextension conda install -y wrapt conda install -y wurlitzer conda install -y xlrd conda install -y xz conda install -y yaml conda install -y yapf conda install -y yarl conda install -y zeromq conda install -y zict conda install -y zipp conda install -y zlib conda install -y zstd
Next create GUI design using lovely prototyping designer: https://www.figma.com/
Open Visual Studio Code and install Flutter extension as follow:
Now create new Flutter app by:
press CTRL+SHIFT+P
and then chose:
In Android Studio create new AVD as follow:
Copy some files to emulator:
Start emulator.
In Visual Studio Code choose Run->Start debugging. Choose emulator as destination.
Your app should look like this:
Now every change in code will be visible on emulator! Simple as it is!
Language Dart is very similar to CSS. It assumes container construction of apps – here is sample of code:
As you can see every element of GUI can be embeded in another one. What is astonishing that in comparison to Java, coding of events like „onPress”, „onTap” is super easy. It is similar to Delphi or even CSS rather than other high-level programming languages.
What is also interesting that Dart allows you to write code simultaneously on two systems: Android and iOS.
Some more interesting info:
All documentation to Flutter and Dart can be found here: https://flutter.dev/docs
When we have our lovely app created – CatsAndDogsClassifier.
It is time to create our model in JupyterLab. Now it will be the most interesting part ;):
To open JupyterLab with chosen disk on Windows-please follow these steps:
open Anaconda terminal and type
conda activate YourEnv
cd D:\YouFolder
jupyter Lab
JupyterLab will be opened then use this notebook.
This notebook is used for classification images acc. to MobileNet model. More info about this model: https://arxiv.org/abs/1704.04861. It recognizes images related to: animals, plants, things.
#import proper libraries import tensorflow as tf import IPython from IPython.display import Image import cv2 import numpy as np from tensorflow.keras.applications import imagenet_utils from tensorflow.keras import backend as K from tensorflow.keras.layers import Dense, Activation #print TensorFlow version=should be 2.1 print(tf.version) # load MobileNet model from internet - it is model trained on 100 000 of images - animals, things, plants model = tf.keras.applications.mobilenet.MobileNet() # it gives overview of used layers in this Net -more info in this book : https://www.amazon.com/Advanced-Deep-Learning-TensorFlow-Keras-ebook/dp/B0851D5YQQ model.summary() # read lables labels = open('mobilenet/labels.txt', 'r') listLabels = labels.readlines() print(listLabels) # load model from the file interpreter = tf.lite.Interpreter(model_path="mobilenet/mobilenet.tflite") # details of used CNN network input_details = interpreter.get_input_details() output_details = interpreter.get_output_details() print(input_details) print(output_details) interpreter.allocate_tensors() # read image, which should be classified from file filename = 'data/cat.jpg' img = cv2.imread(filename) Image(filename=filename) # resize this image-it should have dimensions 224x224 and be in format:uint8 (net was trained on this kind of data) img = cv2.imread(filename) new_img = cv2.resize(img, (224,224)) new_img = new_img.astype('float32') new_img = new_img.astype('uint8') # put image as source data for net interpreter.set_tensor(input_details[0]['index'], [new_img]) interpreter.invoke() # retrieve the result as tensor this model returns several possible unnormalized answers - the most probably is returned as max value from this tensor output_data = interpreter.get_tensor(output_details[0]['index']) output_data[0] from matplotlib import pyplot as plt plt.plot(output_data[0]) # here is the max answer max(output_data[0]) max_index_row = np.argmax(output_data, axis=1) print(max_index_row[0]) output_data[0][0] # here are all possible answers and their unnormalized weights for i in range(len(output_data[0])): if output_data[0][i]>10: print(i) print(output_data[0][i]) print(listLabels[i]) # and finally label for this answer listLabels[max_index_row[0]]
Now it is time for transfer learning. What is transfer learning then?
Transfer learning (TL) is a research problem in machine learning (ML) that focuses on storing knowledge gained while solving one problem and applying it to a different but related problem.[1] For example, knowledge gained while learning to recognize cars could apply when trying to recognize trucks. This area of research bears some relation to the long history of psychological literature on transfer of learning, although formal ties between the two fields are limited.
Approach to Transfer Learning. … The basic premise of transfer learning is simple: take a model trained on a large dataset and transfer its knowledge to a smaller dataset. For object recognition with a CNN, we freeze the early convolutional layers of the network and only train the last few layers which make a prediction.
We will use following notebook:
import tensorflow as tf import tensorflow.keras from tensorflow.keras import Model from tensorflow.keras.models import Sequential from tensorflow.keras.layers import Dense, Input, add from tensorflow.keras.layers import Flatten, Activation from tensorflow.keras.layers import Dropout from tensorflow.keras.layers import AveragePooling1D, AveragePooling2D, MaxPooling2D from tensorflow.keras.layers import Conv1D, Conv2D, BatchNormalization, LSTM from tensorflow.keras.layers import MaxPooling1D, Lambda, Dropout from tensorflow.keras.utils import to_categorical from tensorflow.keras.callbacks import EarlyStopping, ModelCheckpoint from tensorflow.keras.applications import inception_v3 from tensorflow.keras.applications.inception_v3 import InceptionV3 from tensorflow.keras.applications.inception_v3 import preprocess_input as inception_v3_preprocessor from tensorflow.keras.models import Sequential, Model from tensorflow.keras.layers import Dense, GlobalAveragePooling2D, Dropout from tensorflow.keras import optimizers from tensorflow.keras.optimizers import Adam import pandas as pd from pathlib import Path import os.path import os from sklearn.metrics import r2_score from sklearn.model_selection import train_test_split import pandas as pd from scipy.ndimage.interpolation import shift from math import * import matplotlib.pyplot as plt import numpy as np from scipy.fft import rfft, rfftfreq, irfft from scipy.signal import find_peaks from scipy import signal, fftpack from scipy.signal import savgol_filter from sklearn.metrics import mean_squared_error, mean_absolute_error #this is special model used only for transfer learning def create_model():base_model = InceptionV3(weights = 'imagenet', include_top=False, input_shape=(224, 224, 3)) x = base_model.output x = GlobalAveragePooling2D(name='avg_pool')(x) x = Dropout(0.3)(x) x = Dense(4096, activation='relu')(x) x = Dropout(0.3)(x) x = Dense(2048, activation='relu')(x) x = Dropout(0.3)(x) x = Dense(1024, activation='relu')(x) x = Dropout(0.3)(x) x = Dense(512, activation='relu')(x) x = Dropout(0.3)(x) predictions = Dense(1, activation='sigmoid')(x) model = Model(inputs=base_model.input, outputs=predictions) for layer in base_model.layers[:-10]: layer.trainable = False model.compile(Adam(lr=0.00001), loss='binary_crossentropy', metrics=['accuracy'])return model#second version of this model using different loss function def create_model2():base_model = InceptionV3(weights = 'imagenet', include_top=False, input_shape=(224, 224, 3)) x = base_model.output x = GlobalAveragePooling2D(name='avg_pool')(x) x = Dropout(0.3)(x) x = Dense(4096, activation='relu')(x) x = Dropout(0.3)(x) x = Dense(2048, activation='relu')(x) x = Dropout(0.3)(x) x = Dense(1024, activation='relu')(x) x = Dropout(0.3)(x) x = Dense(512, activation='relu')(x) x = Dropout(0.3)(x) predictions = Dense(2, activation='softmax')(x) model = Model(inputs=base_model.input, outputs=predictions) for layer in base_model.layers[:-10]: layer.trainable = False model.compile(Adam(lr=0.00001), loss='categorical_crossentropy', metrics=['accuracy'])return model# Set filepath to the image database images_dir = Path('PetImages/') #Select only .jpg files im_dir = list(images_dir.glob(r'*/.jpg')) imagepath = pd.Series(im_dir, name='Filepath').astype(str) # labels read from folder structure labels = pd.Series(imagepath.apply(lambda x: os.path.split(os.path.split(x)[0])[1]), name='label').astype(str) # Cat detector (if Cat is in the picture = True, else Dog = False) labels2 = labels == 'Cat' # Dataframe images = pd.concat([imagepath, labels2], axis=1).sample(frac=1.0, random_state=1).reset_index(drop=True) image_df = images.sample(5000, random_state=1).reset_index(drop=True) # Training, Validation and Test set train_df, test_df = train_test_split(image_df, train_size=0.7, shuffle=True, random_state=1) train_generator = tf.keras.preprocessing.image.ImageDataGenerator(rescale=1./255, validation_split=0.2) test_generator = tf.keras.preprocessing.image.ImageDataGenerator(rescale=1./255) #it is number of iterations of learning the net epoch_no = 5 #how many images should be provided to net at one iteration batch_s = 8 #extract the train images train_images = train_generator.flow_from_dataframe( dataframe=train_df, x_col='Filepath', y_col='label', target_size=(224, 224), color_mode='rgb', class_mode='raw', batch_size=batch_s, shuffle=True, seed=42, subset='training' ) #extract the validation images val_images = train_generator.flow_from_dataframe( dataframe=train_df, x_col='Filepath', y_col='label', target_size=(224, 224), color_mode='rgb', class_mode='raw', batch_size=batch_s, shuffle=True, seed=42, subset='validation' ) #extract test images test_images = test_generator.flow_from_dataframe( dataframe=test_df, x_col='Filepath', y_col='label', target_size=(224, 224), color_mode='rgb', class_mode='raw', batch_size=batch_s, shuffle=False ) model = create_model() model.summary() #stop learning using best results. It prevents from overfitting cb_checkpointer = ModelCheckpoint(filepath = 'modelinc_ssondl21.hdf5', monitor = 'val_loss', save_best_only = True, mode = 'auto') # train the net history = model.fit(train_images, validation_data=val_images, epochs=epoch_no, callbacks=[cb_checkpointer]) #this saves trained model to file on disk model.load_weights('modelinc.hdf5') #let's check what predictions model returns pred = np.squeeze(model.predict(test_images)) y_test = test_images.labels y_test = y_test.astype(int) from sklearn.metrics import confusion_matrix # this matrix allows to calculate efficiency of net confusion_matrix(y_test, pred.round())
More info about confusion matrix: https://en.wikipedia.org/wiki/Confusion_matrix
Now when we have trained model and saved to disk we can use it further on mobile device.
One more step is needed before using saved model on mobile device-conversion of model to tflite format. To achieve this use this notebook:
import tensorflow as tf
#read model from file hdf5
model = tf.keras.models.load_model('bestInc_skin.hdf5')
#convert it to tflite
converter = tf.lite.TFLiteConverter.from_keras_model(model)
tflite_model = converter.convert()
#save it as .tflite
open("skin.tflite", "wb").write(tflite_model)
Jupyter notebooks are available here.
Here are the steps to use model in mobile app:
And the final result:
a) For cat – probability 98%
b) for unknown probability ~50%
c) dog – probability ~0.02% (probability that it is a cat)
Link exсhange is nothіng ｅlse howeveг it is simply placing the othеr person’s blog link
on your page at proper place and other peгson wіll also do simіlаr in suppߋrt օf you.