An-Najah National University Faculty of Engineering and IT Department of Telecommunication Engineering Graduation Project Report 2 Online Signature Recognition Using CNN &RNN Supervisor: Dr.Khadija Mayyaleh Proposed by: Ekram Othman [11613715] Maram Bani jabber [11611935] Submitted in partial fulfillment of the requirements for Bachelor degree in Telecommunication Engineering 2021/2022 ACKNOWLEDGMENT In the name of Allah, the most gracious, and most merciful. Praise be to Allah, lord of the worlds. My peace and blessing of Allah be upon our prophet Muhammad (SWA) and to his family members, companions and followers. First, We deeply thank and express my sincere gratitude to Allah SWT Who out of HIS infinity mercy and supported us in completing this project. We would like also to express our deep gratitude to our supervisor Dr.Khadija Mayyaleh for her support, guidance and patience throughout our journey, which really makes us work a great learning experience. Form the deep of our heart, we would like also to extend our gratitude to our parents, who strongly supported us in pursuing my educational journey. We also own a large debt to our brothers and sistes, we also want to extend my thanks to Universiti Al-Najah for sponsoring me along our journey. Finally, but not least, we would like to thank all my wonderful friends and to everyone supported us to complete this project journey. Disclaimer statement This report was written by students at the Telecommunication Engineering Department, Faculty of Engineering, An-Najah National University. It has not been altered or corrected, other than editorial corrections, as a result of the assessment and it may contain language as well as content errors. The views expressed in it together with any outcomes and recommendations are solely those of the students. An-Najah National University accepts no responsibility or liability for the consequences of this report being used for a purpose other than the purpose for which it was commissioned. Contents ACKNOWLEDGMENT ii Disclaimer statement iii List of Figures (LOF) v List of Tables (LOT) vi Abstract vii Chapter 1: Introduction 12 Chapter 2: Constraints, Standards/ Codes and Earlier course work 15 Symmetrical encryptin 15 Asymmetric encryption 16 Online banking 17 Feature Extraction 18 PSF Features 19 Temporal Enhanced PSF Features 19 Chapter 3: Literature Review 21 Chapter 4: Methodology 26 First Part: Online Collect Signature 26 Second Part: Deep Learning Methods 31 Numerical Experiments 34 Dataset 34 Experimental Settings 35 Chapter 5: Results and Analysis 37 Chapter 6: Discussion 40 Chapter 7: Conclusions and Recommendation 41 References 42 List of Figures (LOF) Figure 1 Principle of symmetrical encryption 15 Figure 2Principle of asymmetric encryption 16 Figure 3 Signature as a sequence of stroke points 18 Figure 4Comparison of two kinds of PSF 20 Figure 5 Example of signature model for each user 23 Figure 6 Global Definition 27 Figure 7 Mouse Event 30 Figure 8 : LeNet-5 Structure 32 Figure 9 Different Signatures and Different Widths 32 Figure 10 RNN Structure 33 Figure 11 CNN-LSTM Architecture 34 Figure 12 Learning Rate Selection Process 35 Figure 13 Loss Value in CNN Experiments 38 List of Tables (LOT) Table 1 . List of common features 22 Table 2 Architecture Configuration of CNN-LSTM Model 34 Table 3 Classification results on testing dataset of varying model architectures 36 Abstract Signature is widely used in human daily lives, and serves as a supplementary characteristic for verifying human identity. However, there is rare work of verifying signature. In this paper, we propose a few deep learning architectures to tackle this task, ranging from CNN, RNN to CNN-RNN compact model. We also improve Path Signature Features by encoding temporal information in order to enlarge the discrepancy between genuine and forgery signatures. Our numerical experiments demonstrate the effectiveness of our constructed models and features representations, also the experimental results indicate significant error reduction and accuracy enhancement in comparison with state of the art counterparts. i Chapter 1: Introduction Nowadays, signatures play an essential role in people's daily lives, and they are used as a supplemental feature to aid identity verification in a variety of situations, such as credit card fraud and criminal investigation [1]. Signature forgery, on the other hand, develops quickly and significantly degrades people's possessions. Signature forgery has even become a lucrative illicit business in some areas. On EBay, services for creating false signatures are available [2]. Fake signatures are used by certain persons to swindle others' assets and to bilk huge organizations like insurance companies and banks. Since the real estate bubble burst in 2006, fraudulent signatures on papers have resulted in a $25 billion settlement between lenders, governments, and most states, which is expected to last through 2012 [3]. In 2019, the National Crime Agency and the Financial Conduct Authority in the United Kingdom were requested to look into allegations that bank staff faked signatures on court documents to reclaim debts and properties, which is a scandal [4]. It is critical to develop an effective signature verification system that can distinguish real signatures from forgeries in order to avoid exploiting forging signatures. The difficulties in attaining successful signature verification can be divided into two categories: Intrapersonal variability is considerable, but interpersonal variance is minimal. The former alludes to the fact that a single person can sign in a variety of ways depending on the situation. The latter implies that if a forger has seen a real signature, he or she may create one with identical geometric qualities very quickly. The difficulties mentioned above make accurate verification more difficult. To address these issues, a state-of-the-art signature verification technique employs the discrete wavelet transform (DWT) to increase the difference between genuine and forged signatures, followed by a simple feed forward neural network (consisting of stacking fully connected layers) to perform binary classification [5]. However, the characteristic retrieved by DWT is computationally demanding and difficult to implement properly, which prevents it from being used in other countries. Deep learning, on the other hand, has shown extraordinary effectiveness in a variety of scientific applications [6], spanning from computer vision to handwriting [7]. Convolutional Neural Network (CNN) and Recurrent Neural Network (RNN) have showed promise in handwritten regions, based on a variety of decorated feature extractions like as Bezier curves and Path Signature Features [8]. Traditional pattern recognition approaches have been found to be inferior to convolutional neural networks (CNN) and recurrent neural networks (RNN). As a result, it's only natural to wonder if an effective signature verification system can be built using recent deep learning architectures. To solve this topic, we suggest various methods for developing signature verification systems using recent deep learning techniques in this study. Our major contributions are summarized in the bullet points below. · We construct and explore shifting present day profound learning strategies to attain successful signature confirmation, extending from CNN, RNN to CNN-RNN compact models. Our developed models are simple to execute and send comparing with state-of-the-art DWT approach. And we numerically illustrate the viability of our proposed strategies for signature verification. · We progress the Way Signature Highlight by encoding transient data into it to open up the error within the worldly perspective to resolve the issue of high interpersonal change of marks. This paper is organized as follows. At first, in Section 2, we present how collect the signature online and feature extraction works. Then we move on to describe the deep learning model architecture candidates in Section 3. Next, in Section 4, we numerically test and compare the performance of our built models, and finally conclude this paper in Section 5. Chapter 2: Constraints, Standards/ Codes and Earlier course work Constraints: Some of the limitations that we faced in completing our second graduation project are learning completely new programming languages ​​for us, such as Python and JavaScript in an advanced way. Also, the application was working to implement a new signature recognition online process on us. Standards/Codes: -Python -JavaScript -T-Script -application in HTML -CNN -RNN Earlier course work: -advanced programing(java) -C++ -Advanced lab -information theory -speech and audio processing projects Symmetrical encryptin Figure 1 Principle of symmetrical encryption Original document Sender encrypted with a secret key Encrypted Recipient decrypts with a secret key Original document A disadvantage of this method is that the sender and recipient have to agree on the secret key. In practical terms, this means that, for example, when communicating by e-mail, the sender and recipient have to agree on the secret key by telephone. The only and secret key must therefore not get into third hands and must be kept safe. Symmetrical encryption cannot be used to create a signature. Since the same key must always be in the possession of several people, it can never be reliably determined which of them encrypted a document. For example, if three people have the same symmetric key and they encrypt the same document, the three encrypted documents do not differ from one another. It is therefore not possible to trace the documents back to one of the three people. Asymmetric encryption These shortcomings can be eliminated by the principles of asymmetric encryption. With asymmetric encryption there is a key pair that consists of a secret part (private key) and a non-secret part (public key). This public key can be communicated to any communication partner, for example it can also be published on a website. The private key, on the other hand, like the secret key with symmetrical encryption, must be kept safe and must not be passed on to others. Original document Sender encrypted with the recipient's public key Encrypted Recipient decrypts with his private key Original document Figure 2Principle of asymmetric encryption The private key enables its owner, for example, to decrypt data (see Figure 2), to generate digital signatures or to authenticate himself. The public key enables anyone to encrypt data for the key holder (see Figure 2), to check their digital signatures or to authenticate them. In contrast to a symmetrical encryption system, the communicating parties do not need to know a shared secret key. It is therefore not possible to decrypt a document that was encrypted with the public key again with the public key. Such a document could only be decrypted again with the private key. Conversely, a document encrypted with the private key can only be decrypted again with the public key. The digital signature makes use of this principle. A document (or more correctly a mathematical checksum of a document) is encrypted with the secret private key. If you now know who is in possession of the private key, you can use the public key (which is freely accessible on the Internet) to check whether this person has actually encrypted the corresponding document: the document can be read again with the public key this document must be produced by the person who is the only one who has the private key and who originally encrypted it. Online banking The digital signature is also suitable for the secure execution of bank transactions. For some time now, malware has been reported as a threat in online banking. For example, there are Trojan horses that intercept entered transaction numbers and misuse them for unintended solutions by the account holder. Such malicious programs typically get stuck in their victim's browser. The use of electronic signatures can provide effective protection against such attacks. Numerous banks offer online banking based on electronic signatures, whereby the orders to the bank are provided with the customer's secure electronic signature. In order to be able to create secure electronic signatures, you need a chip card reader with its own PIN keypad that prevents the PIN from being spied on. The use of suitable software components (Secure Viewer) ensures that the data to be signed is presented to the signer (authorized signatory), e.g. on the monitor, before the signature is created, and changes after the signature has been created can be recognized. Malicious programs that get stuck in the browser have no effect on Secure Viewer. You can neither create nor change signed orders without the assistance of the customer. In addition, most secure viewers are subjected to a strict check in accordance with recognized security specifications, whereby, among other things, the immutability of the data to be signed must be proven. Feature Extraction Feature Extraction Raw signature instances, essentially handwritten texts, are collected by recording the motions of digital pens on a tablet. They can be regarded a sequence of strokes which consist of a series of stroke points, as shown in Figure 1. For a more rigorous description, denote S = {s1, s2, …. , sn} as a handwritten signature instance, making up by a sequence of strokes, where si refers as the i-th stroke in current handwritten signature instance. One stroke s comprises a set of sequential stroke points, i.e., s = {(x1, y1, p1, t1),(x2, y2, p2, t2), … }, where (xj , yj , pj , tj ) is j-th stroke point with (xj , yj ) as its coordinate axis, pj ∈ {0, 1} as pen up/down status to indicate the starting or ending stroke point, and tj as timestamp. Figure 3 Signature as a sequence of stroke points Because the target signature data is sequential, a recurrent neural network looks like an obvious choice for the verification task. However, with different underlying ink representation techniques, digital ink representations of the same handwritten text, particularly the amount of stroke points, might vary substantially, resulting in considerable generalization discrepancies. Rendering stroke points into binary images appears to be a plausible solution to this problem, but it loses a lot of essential information during the rendering process, such as the order and evolution of stroke points. We apply Path Signature Features (PSF), which has been shown to be effective in online handwritten text recognition, to balance rendering technique and sequential property of handwritten data. PSF Features Path signature feature is defined over a path of stroke points, and usually calculated up to 2nd order [8, 9]. More specifically, for each smallest stroke point path {(xi , yi),(xi+1, yi+1)} consisting of 2 consecutive points, the signature feature pi,i+1 = (p (0) i,i+1, p (1) i,i+1, p (2) i,i+1) ∈ R 7 is calculated as follows: where ⊗ represents Kronecker matrix product. It follows the definition of PSF (1) that PSF encodes geometrical and order information of stroke points into seven path signature maps to form a 3D tensor T ∈ R 7×H×W , where H and W represent the height and with respectively. As [8, 9], we fix the height H as 128, and let width be flexible. Temporal Enhanced PSF Features The original PSF features in form of (1) only encodes spatial information and order of stroke point into 3D tensor, which has been demonstrated its utility in Handwritten Recognition and Synthesis domain [8]. However, remark here that the original PSF does not consider any temporal information of digital ink, e.g., the speed of pen movement, which may be crucial for many other applications. Particularly, for our target signature verification, the temporal information may be beneficial to achieve a better generalization performance, because a forger may spend longer time to mimic one genuine signature. Inspired by it, to incorporate temporal information, we design an enhanced PSF features as follows: where a coefficient τ is computed as the logarithm of i-th stroke point’s timestamp, then each PSF feature point is multiplied by this coefficient correspondingly. By utilizing such modified PSF features, the temporal information is involved as illustrated in Figure 2. The effectiveness of such temporal enhanced PSF will be verified later. Figure 4Comparison of two kinds of PSF Earlier coursework: In this part, the student should reflect any earlier coursework and topics used in the project. This can be done by stating how earlier coursework has been utilized in doing the project effectively. Chapter 3: Literature Review Most recent approaches in the field of online signature verification have been described in [14, 15, 33]. The process of signature verification is usually divided into three phases: 1. Pre-processing The signature dataset must take some pre-processes since there is no guarantee that different signatures of one user will always be the same. Several processes have been proposed for this phase, which generally consist of smoothing, rotation and normalization. Cubic splines can be employed for smoothing purpose to solve the jaggedness in the signatures. Signatures can become rotation-invariant by rotating each one based on orthogonal regression (Eq. 3) [15]. (3) Where sx and sy are variance and cov x, y is covariance of the horizontal and vertical components. The signatures of one person must have the same size for better performance. The horizontal and vertical components of signatures can be normalized to make a standard size(Eq.4). (4) 2. Feature Extraction Feature selection and feature extraction play an important role in verification systems. Many studies have been performed in the field of feature selection to choose the best set of features for extraction. List of common features have been described in Table 1 [15]. Table 1 . List of common features Furthermore, some non-common features have been described in other papers [1-3, 23, 24, 27, 31]. Recently, instead of handcrafted features in traditional biometric authentication systems for face, iris and fingerprint, some more discriminative features which are provided using deep learning are exploited [22, 30]. 3. Classification After the feature extraction phase, the system must create a model from reference signatures. For classification phase, each signature must be compared against reference signatures and by calculating the distances between test and reference signatures, the system can decide to accept or reject the test signature. As mentioned, in daily usage of authenticating systems such as banking systems, handwritten signature of users has been used to verify the identity of official documents. In these sets of problem, the main goal is verifying whether a signature belongs to one identified person or not. In contrary to multiclass classifiers, the aim for one-class classifiers is ETRI Journal, Volume x, Number y, Date First Author et al. 3 distinguishing one type of class (target) from other classes (outlier). Thus, for classifying a signature as genuine or forgery, one-class classifiers have been commonly used [22] to divide the set into two categories: target and outlier (Figure 5). As it shown, detecting the random forgery ones is very easier than skilled forgery ones. Figure 5 Example of signature model for each user Jain and Gangrade [17] proposed a system by using angle, energy and chain code features to differentiate the signatures. In this approach, a Neural Network has been applied for classification. Faundez-Zanuy [6] studied four pattern recognition algorithms for online signature recognition: Vector Quantization (VQ), Nearest Neighbor, Dynamic Time Warping (DTW) and Hidden Markov Model (HMM). The author proposed two methods based on VQ and Nearest Neighbor. Barkoula, et al. [3] studied the signatures Turning Angle Sequence (TAS), the Turning Angle Scale Space (TASS) representations, and their application to online signature verification. In the matching stage, the authors have employed a variation of the longest common sub-sequence matching technique. Rashidi, et al. [15] evaluated 19 dynamic features viewpoint classification errors and discrimination capability between genuine and forgery signatures. They used a modified distance of DTW for improving performance of verification phase. Ansari, et al. [2] presented an online signature verification system based on fuzzy modelling. The point of geometric extrema has been chosen for signature segmentation and a minimum distance alignment between samples has been made by DTW techniques. Dynamic features have been converted to a fuzzy model and a user-dependent threshold used for classification. Yahyatabar, et al. [18] proposed a method based on efficient features defined in Persian signatures. A combination of shape based and dynamic extracted features has been applied and a SVM has been used for classification phase. Alhaddad, et al. [1] explored a new technique by combining back-propagation Neural Network (BPNN) and the probabilistic model. BPNN has been used for local features classification, while probabilistic model has been used to classify global features. Mohammadi and Faez [23] proposed a method based on the correspondence between important points in the direction of wrap for the time signal provided to maximize the distinction between the genuine and forged signatures. Napa and Memon [24] Presented a simple and effective method for signature verification in which an online signature is represented with a discriminative feature vector derived from attributes of several histograms that can be computed in linear time. For testing phase, the authors proposed a method on finger drawn signatures on touch devices by collecting a dataset from an uncontrolled environment and over multiple sessions. Souza, et al. [22] proposed an off-line signature verification system, which uses a combination of five distance measurements, such as; furthest, nearest, template and central using four operations: product, mean, maximum, and minimum as a feature vector. Fallah, et al. [5] presented a new signature verification system based on Mellin transform. The features have been extracted by Mel Frequency Cepstral Coefficient (MFCC). Neural Network with multi-layer perception architecture and linear classifier in conjunction with Principal Component Analysis (PCA) have used for classification. Iranmanesh, et al. [16] proposed a verification system by using multi-layer perceptron (MLP) on a subset of PCA features. This approach used a feature selection method on the information that has been discarded by PCA, which significantly reduced the error rate. Cpałka, et al. [4] explored a new method by using area partitioning of high and low speed of the signature and high and low pen's pressure. The template for each partition has been generated and by calculating the distance between signatures and template in each partition, a fuzzy classification has been implemented to classify the signatures. Lopez-Garcia, et al. [21] presented a signature verification system implemented on an embedded system. In this approach, a template for each user has been generated and a DTW algorithm has been used for distance calculation. Finally, the features extracted and passed through a Gaussian Mixture Model (GMM) to calculate the similarity between the test signature and the generated template. Chapter 4: Methodology First Part: Online Collect Signature We design an application for collect signature, it is a HTML form into a signature box. The Signature box has two modes: Type and Draw. In Type mode the user’s signature is automatically generated as HTML text, from the input field where they type their name. In Draw mode the user is able to draw their signature on the canvas element. The drawn signature is written out to a hidden input field as a JSON(JavaScript Object Notation) array. Since the signature is saved as JSON it can be submitted as part of the form and kept on file. Using the JSON array, the signature can then be regenerated into the canvas element for display. The application works with both mouse and touch devices. Basic usage of canvas At first sight a  looks like the  element, with the only clear difference being that it doesn't have the src and alt attributes. Indeed, the  element has only two attributes, width and height. These are both optional and can also be set using DOM properties. When no width and height attributes are specified, the canvas will initially be 300 pixels wide and 150 pixels high. The element can be sized arbitrarily by CSS(Cascading Style Sheets is a style sheet language used for describing the presentation of a document written in a markup language such as HTML), but during rendering the image is scaled to fit its layout size: if the CSS sizing doesn't respect the ratio of the initial canvas, it will appear distorted. Customize Cnaves to our needs in 7 steps. 1. Canvas Element Cnaves elements are size and id . All drawings of the chart will be put on it by using Javascript objects. 2. Global Definition Figure 6 Global Definition Initially, using getElementById() with id will get an object canvas, and then there are 3 essential properties, canvasW, canvasH, and ctx(context), to be retrieved from it. As line chart is flat, here we set ctx to be a two-dimension canvas. margin means the distances of 4 directions between drawings and canvas boundaries. Based on it, we can calculate chart width chartW and chart height chartH. origin stands for a base point in the coordinate system. The horizontal x-axis and vertical y-axis cross at origin according to mathematical definition. From the view of coordinate system, key coordinates such as origin, x_axis_end, and y_axis_end should be found out. To calculate 3 coordinates for top, bottom, and left titles of this canvas chart. The right side of the canvas chart has no labelled title, but a legend, which usually describes each set of data in a chart. At last, we find out the legend coordinate at right side. 3.  Rendering Line Chart To draw, convert input data into JSON style on HTML loading, and then invoke the method lchart.render() of a Javascript object lchart. 4. Basic Functions for Point, Line, Text The ldraw.point() method can produce points in awesome gradiant colors by using object grad from ctx. In particular, the Javascript object ctx = canvas.getContext("2d"). Unlike, the ldraw.text() method writes text in HTML5 Canvas charts in a more simple way without enclosure. 5. Coordinates Transformation HTML5 Canvas is positioning Y values of coordinates from top to bottom, while line charts have a coordinate system contrary to that. To draw on  elements, you need to convert from coordinate of line chart to that of HTML5 Canvas. 6. Sketching X-AXIS & Y-AXIS One thing you should realize before drawing the chart is the sequence or saying particular order. Every drawing in HTML5 Canvas overlays each other to result in one image, rather than plenty of vectors in graphics. The latter picture will always cover the previous one, so chart background with the size decided by canvasW and canvasH should be drawn first. To sketch Axises means not only to draw axis lines, but also to create axis unit marks and axis unit labels. Javascript object ltrans provide two methods convert() and adjust() to properly transform and calibrate related coordinates, respectively. Moreover, there are 3 titles on the positions of top, left, and bottom to display. 7.  Depicting Points & Legend To create pictures on a canvas chart, lines were always sketched before drawing points and texts, because of the pixel overlay feature as mentioned in Step 6. These coordinates should be transformed by ltrans.convert() or adjusted by ltrans.adjust() before drawing. About lines and points coloring, Step 2. Global Definition has specified colors in legend_colors that can paint both legend and chart elements. Signatures Process inside the box I. Add signature 1. Listen to Mouse Events In general, an event-driven model fits how signatures are created to our canvas. For example, the event E: mouse down make a state transitionzqa from IDLE state to DRAW state. Along with state transition, the action A: set start_pos sets starting position of a line segment to be sketched. Figure 7 Mouse Event Then when user start hand drawing inside the box corresponds to this state transition diagram. Accordingly, there are 3 canvas methods addEventListener() listening to events of Event mouse down, Event mouse move, and Event mouse up. Once users operate on mouse, events will trigger both changing state and performing actions if there exists. Often, a signature curve consists of many connected line segments on HTML5 Canvas. As in the program, when a mousemove event occurs and current state is on_drawing, the procedure will be locating an end point, drawing a line segment, and setting a new start point. mousemove events generate a lot of connected line segments which constitute a hand-writing curve. Once a mouseup event happens in DRAW state, the state becomes IDLE. At this time, any mousemove event won’t contribute to the signature on canvas.   2. Get Mouse Positions & Draw Line Segments Knowing how to locate mouse cursor is a little bit complex. In short, both the canvas method getBoundingClientRect() and the callback function of canvas method addEventListener() provide two pairs of X-Y coordinates, rect and evt. The distances of them are what you want. Clear Signature ctx = canvas.getContext("2d"). It supports most operations on canvas such as clearing content and drawing line segments. Save Canvas Signature · Download as image Javascript method hdwrite.download() can download the signature on HTML5 canvas to be an image file of PNG or JPG type or file.. Save to database and use our online algorithm for classification. HTML APPLICATION Example of drawn signature in HTML Second Part: Deep Learning Methods In this section, we present varying deep learning model architectures that utilizes the PSF features for signature verification. • Convolutional Neural Networks The PSF features are essentially a 3D tensor, i.e., T ∈ R 7×H×W. To handle such 3D tensors, Convolutional neural network (CNN) is a natural choice which have demonstrated its remarkable success in plentiful computer vision and handwritten applications [10]. Among numerous popular CNNs, we make use of LeNet-5 [11], a classic CNN architecture for handwritten character recognition as the basic model, and adjust its structure properly to feed PSF tensors and binary classification as shown in Figure 8 Figure 8 : LeNet-5 Structure Since the different signatures possess varying lengths, which causes the extracted PSF tensors also own different widths, see Figure 9. The different widths further result in the discrepancy on input sizes of fully connected layers. To address this problem, in the basic CNN model, we scale PSF tensors to fix the width as its height, namely 128 Figure 9 Different Signatures and Different Widths · Recurrent Neural Network Signatures consist of a sequence of strokes, of which is composed of a sequence of stroke points. However, RNN has a drawback of overflow as the length of input sequential data is enormous. Figure 10 RNN Structure · CNN-LSTM Compact Model Scaling and fixing the width and height of PSF tensors has the drawback that breaks down the original geometrical property of signature data. At first, the seven path signature maps extracted as described above are stacked to form a seven-channel PSF tensor denoted as S, which serves as the input of the following CNN-LSTM binary classification model, as illustrated in Figure 11. CNN serves to encode extracted path signature feature into a matrix, since the input 3D tensor may possess varying widths, consequently results in that the output matrix of CNN possesses varying columns. then each column of this encoded matrix is further fed into an LSTM(Long Short-Term Memory ) cell sequentially, an FNN is assembled on the top of last output of the LSTM to calculate the probability of current instance as genuine written text. The whole procedure is shown in Figure 11. Figure 11 CNN-LSTM Architecture Table 2 Architecture Configuration of CNN-LSTM Model Where input shape, kernel shape and output shape are in the manners of (width, height, channel), (# of kernels, mask, stride) and (width, height, channel) respectively. The stride can either be a single number for both width and height stride, or a tuple (sW, sH) representing width and height stride separately. Numerical Experiments In this section, we present and discuss experimental results of the constructed models in previous section to numerically demonstrate the effectiveness of some built model candidates.then we use our data set from our collection of signeatures. Dataset The training and testing datasets is selected as the public benchmark signature data, namely SVC2004 and our experimental data set from our collection signeatures ex: group members signatures .These signature samples are collected through digital input boards and related equipments, not only providing signature information but also collecting information related to intrapersonal features, such as writing speed and pressure. Each signature instance is represented as a series of stroke points. Each stroke point consists of several features, i.e., index, x and y coordinate, timestamp, pen up/down status, azimuth, altitude, and pressure. Data sets Signeature drawn example in the app Experimental Settings In the deep learning and artificial intelligence part, we wrote Python codes for CNN & RNN in the classification stage, where we used a combination of CNN_RNN together to achieve perfect accuracy, speed and performance for the signature matching verification phase. After collecting and drawing signatures using the signatures application, we ran the special Python codes in the verification phase and uploaded these signatures to the design of the verification process as shown in the design: Design signature validation based on python codes. The system requires us to enter only two signatures for the confirmation process, we draw two signatures by the same person on the previously designed HTML application and test the extent of their matching. It turned out that both signatures were for the same person and the verification process was successful. First signature result 100% recognition because the two drawn signature was typically the same. Second: another two signature in different accuracy 97.13% Third : recognition rate is 96.01%. The system asks us to enter two signatures only for the confirmation process, we drew two signatures by the same person on the previously designed HTML application, but in two different ways, and the difference was not significant, and we tested the extent of their conformity. The system recognized this simple difference and rejected the verification process. The mini-batch size of our experiment is set as 10. Figure 12 Learning Rate Selection Process Learning rate plays a vital role in the deep learning applications. In order to find an optimal initial learning rate, we use the method in [12]. Then, looking for the lowest point of loss, and select the initial learning rate as the one corresponding to the lowest loss function value, as shown in Figure 12. Such initial learning rate selection mechanism helps our training process a lot. After determining it, we decay the learning rate exponentially per epoch by a ratio as 0.95. Table 3 Classification results on testing dataset of varying model architectures Chapter 5: Results and Analysis We now summarize the generalization performances of varied deep learning model architectures on testing dataset in Table 3. To facilitate comparison, the best values are marked as red, and the second best values are marked as blue. The “old PSF” refers the original PSF in form of (1), the “new PSF” refers to temporal enhanced PSF as (2), and the “stacked PSF” means a tensor formed by stacking original PSF and temporal enhanced PSF along channel axis. 1. CNN with original PSF: As shown in Table 3, CNN with original scaled PSF features as input cannot achieve satisfactory classification performance. More specifically, as illustrated in Figure 8, after numerous training epochs, the value of loss function converges to a high value, namely 0.693. Such bad convergence results in a 51.4% accuracy, high recall but low precision on testing datasets, which means that the model tends to judge most of the data points as positive instances. This result also indicates that even though PSF in form of (1) serves as very effective features for handwritten recognition, it helps limited in signature verification 2. CNN with enhanced PSF: After improving the PSF by incorporating temporal information into the tensor entries, we feed such enhanced PSF into the same CNN structure, and observe a significant improvement shown in Table 2. Particularly, regarding the evolution of loss function values, we can see as the number of epochs increases, the loss function values keeps decreasing generally and stalks around 0.01. On the other hand, the accuracy also rises to 73.9%, we relatively high recall as 76.2% and precision as 72.8%. It is a strong evidence that our temporal enhanced PSF works well to encode and enlarge the temporal discrepancy between genuine and forgery signatures. Figure 13 Loss Value in CNN Experiments 3. RNN: Before employing RNN on signature verification, we at first resample the stroke points of each signature instance to let them share the same number of stroke points by uniform sampling algorithm in [14] with some modifications. LSTM is selected the RNN cell, and then we can achieve classification metrics around 60-70%. 4. CNN-RNN Compact Model with original PSF: We then use the CNNLSTM model to allow original PSF tensors with varying widths. As shown in Table 3, the performance of the CNN-LSTM model exceeds the ordinary CNN a lot, and is slightly better than the RNN model. The accuracy, precision, recall and F1 score increase to 70.0-80.0%, from which we can see CNN-LSTM architecture is a more effective topological structure to achieve better generalization performance. 5. CNN-RNN Compact Model with enhanced PSF: Previous experiments have revealed that both the temporal enhanced PSF and CNN-LSTM structure are beneficial for signature verification. Therefore, we expect that the combination of the improved PSF and CNN-LSTM models may achieve more outstanding results. The experimental results confirmed our hypothesis that this accuracy rate rose to 99.7% _ 100 % which is the best result so far. 93% as a worst case recognition using CNN_RNN. 100% as a standard case of recognition . Chapter 6: Discussion The result indicates that even though PSF in form of (1) serves as very effective features for handwritten recognition, it helps limited in signature verification The reason is because of the low interpersonal variance, i.e., if someone is aware of your signature ahead, he/she can probably mimic a very similar forgery signature in terms of spatial property. Due to the high spatial similarity, the original PSF as (1) does not possess the capability to distinguish the discrepancy. Therefore, the original PSF is not an effective feature for signature verification, and needs to be improved. This leads to our subsequent algorithm improvement and the next series of experiments. Chapter 7: Conclusions and Recommendation In this paper, we tried a few modern deep learning methods to achieve signature verification, including CNN, RNN and CNN-RNN compact architectures. We make use of Path Signature Feature as input to encode spatial information of signature instances, and improve original PSF with incorporated temporal information to resolve the issue of low interpersonal variance. Our numerical experiments demonstrate the effectiveness of our model architecture and improved PSF features. This method has proved its ability to extract the best set of features in problems that need to define hand-crafted features. Therefore, it can be used in a wide range of machine learning problems. As a future work, this method can be tested on offline signatures. In addition, the impact of deep convolutional networks can be tested on both online and offline signature datasets. References [1] D. Impedovo and G. Pirlo, "Automatic Signature Verification: The State of the Art," Systems, Man, and Cybernetics, Part C: Applications and Reviews, IEEE Transactions on, vol. 38, pp. 609- 635, 2008. [2] D. Impedovo, G. Pirlo, and R. Plamondon, "Handwritten Signature Verification: New Advancements and Open Issues," in Frontiers in Handwriting Recognition (ICFHR), 2012 International Conference on, 2012, pp. 367-372. [3] D. Menotti, G. Chiachia, A. Pinto, W. Schwartz, H. Pedrini, A. Falcao, et al., "Deep Representations for Iris, Face, and Fingerprint Spoofing Detection," Information Forensics and Security, IEEE Transactions on, vol. 10, 2015. [4] M. Faundez-Zanuy, "On-line signature recognition based on VQDTW," Pattern Recognition, vol. 40, pp. 981-992, 3// 2007. [5] S. Rashidi, A. Fallah, and F. Towhidkhah, "Authentication based on signature verification using position, velocity, acceleration and Jerk signals," in Information Security and Cryptology (ISCISC), 2012 9th International ISC Conference on, 2012, pp. 26-31. [6] A. Q. Ansari, M. Hanmandlu, J. Kour, and A. K. Singh, "Online signature verification using segment-level fuzzy modelling," Biometrics, IET, vol. 3, pp. 113-127, 2014. [7] P. Jain and J. Gangrade, "Online Signature Verification Using Energy, Angle and Directional Gradient Feature with Neural Network," International Journal of Computer Science and Information Technologies (IJCSIT), vol. 5, pp. 211-216, 2014. [8] J. Fierrez, J. Ortega-Garcia, D. Ramos, and J. Gonzalez-Rodriguez, "HMM-based on-line signature verification: Feature extraction and signature modeling," Pattern Recognition Letters, vol. 28, pp. 2325-2334, 12/1/ 2007. [9] K. Barkoula, G. Economou, and S. Fotopoulos, "Online signature verification based on signatures turning angle representation using longest common subsequence matching," International Journal on Document Analysis and Recognition (IJDAR), vol. 16, pp. 261- 272, 2013/09/01 2013. [10] Z. Zhang, K. Wang, and Y. Wang, "A Survey of On-line Signature Verification," in Biometric Recognition. vol. 7098, Z. Sun, J. Lai, 10 X. Chen, and T. Tan, Eds., ed: Springer Berlin Heidelberg, 2011, pp. 141-149. [11] M. E. Yahyatabar, Y. Baleghi, and M. R. Karami, "Online signature verification: A Persian-language specific approach," in Electrical Engineering (ICEE), 2013 21st Iranian Conference on, 2013, pp. 1-6. [12] M. H. Mohammadi and K. Faez, "Matching between Important Points using Dynamic Time Warping for Online Signature Verification," Cyber Journals: Multidisciplinary Journals in Science and Technology, Journal of Selected Areas in Bioinformatics (JBIO), 2012. [14] S.-B. Napa and N. Memon, "Online Signature Verification on Mobile Devices," Information Forensics and Security, IEEE Transactions on, vol. 9, pp. 933-947, 2014. [15] A. Kholmatov and B. Yanikoglu, "SUSIG: an on-line signature database, associated protocols and benchmark results," Pattern Analysis and Applications, vol. 12, pp. 227-236, 2009/09/01 2009. [16] R. Wang, C. Han, Y. Wu, and T. Guo, "Fingerprint Classification Based on Depth Neural Network," The Computing Research Repository (CoRR), vol. September 2014, 2014. [17] M. R. P. Souza, G. D. C. Cavalcanti, and R. Tsang Ing, "Off-line Signature Verification: An Approach Based on Combining Distances and One-class Classifiers," in Tools with Artificial Intelligence (ICTAI), 2010 22nd IEEE International Conference on, 2010, pp. 7-11. [18] M. T. Ibrahim, M. Kyan, and G. Ling, "On-line signature verification using global features," in Electrical and Computer Engineering, 2009. CCECE '09. Canadian Conference on, 2009, pp. 682-685. [20] K. Cpałka, M. Zalasiński, and L. Rutkowski, "New method for the on-line signature verification based on horizontal partitioning," Pattern Recognition, vol. 47, pp. 2652–2661, 2014. [21] M. Lopez-Garcia, R. Ramos-Lara, O. Miguel-Hurtado, and E. Canto-Navarro, "Embedded System for Biometric Online Signature Verification," Industrial Informatics, IEEE Transactions on, vol. 10, pp. 491-501, 2014. [22] M. Fayyaz, M. H. Saffar, M. Sabokrou, M. Hoseini, and M. Fathy, "Online Signature Verification Based on Feature Representation," in International Symposium on Artificial Intelligence and Signal Processing, Iran, Mashhad, 2015. [23] H. Song and S.-Y. Lee, "Hierarchical Representation Using NMF," in Neural Information Processing. vol. 8226, M. Lee, A. Hirose, Z.-G. Hou, and R. Kil, Eds., ed: Springer Berlin Heidelberg, 2013, pp. 466-473. [24] R. Raina, A. Battle, H. Lee, B. Packer, and A. Y. Ng, "Self-taught learning: transfer learning from unlabeled data," presented at the Proceedings of the 24th international conference on Machine learning, Corvalis, Oregon, USA, 2007. image3.png image4.png image5.png image6.png image7.png image8.png image9.png image10.png image11.png image12.png image13.png image14.png image15.png image16.png image17.png image18.png image19.png image20.png image21.png image22.png image23.png image24.png image25.png image26.png image27.png image28.png image29.png image30.png image31.png image32.png image33.png image1.png image2.png