Visual system for punched card reading in textile industry

In this paper we present a novel design and implementation of a vision-based system for reading punched cards. The new system is designed to visually retrieve information stored in punched cardsandstorethemindigitalformat.Thesystemisdedicatedtotypesofcardsusedinmechani-calJacquardloomswidelyspreadindifferenttextileindustries.Imageprocessingtechniqueslike searchingforedgesusinggradient,normalizedcrosscorrelation,andimageregistrationareeffi-cientlyutilizedtofaceseveralchallenges.Theexistenceofdifferenttypesofdisturbancesmakes gradient-basedsearchforthepatternholesinsufficienttoobtainzeroerrorratio.Asexpectedimageregistrationcancopewithsuchaproblemafterefficientlysolvingthefeatureextraction andmatchingproblemresultedfromthefactthatthecardimagedoesnotcontainanykindofinformationexceptthepatternholeswhicharenotdiscriminatingfeatures.Experimentshave beencarriedoutusingthebuiltrealsystem.Experimentsshowresultswith0.041%errorratio.Thisratioishighlysatisfyingthetextileindustry.


Introduction
Machine vision has been demonstrated to be successful for a wide range of robotics and industrial applications [1][2][3][4]. The revolution that was made by introducing the automation to industrial applications like manufacturing, assembly, and inspection raised the needs for machine vision systems that provide higher efficiencies than the human workers [5]. Vision sensors have recently become an essential module in most of industrial and manufacturing areas [6][7][8][9][10]. Particularly, textile industry has so many successful vision applications.
Visual technology is used in different stages of the manufacturing process of textile production [6]. Consequently, different visual technology tools are used as needed to accomplish the production tasks. The system presented in [11] uses Gabor filter for defect detection in textile fabrics. A carefully designed, based on texture's features of non-defected fabric, Gabor filter is used along with smoothing filter. The system presented in [4] controls the quality of textile fabrics. The used dedicated hardware design enables the system to work in presence of many types of industrial noise. The result is a powerful system for defect inspection and detecting so many kinds of fabric defects under real industrial conditions. The vision algorithm use texture elements as input to a neural network algorithms.
This paper mainly aims at designing and building a visual card reader of Jacquard punched cards for textile machines. Visual technologies are an excellent and cheap means to build such a system. A camera connected to a computer and a motor to rotate the film of punched cards are the main components of the proposed system. However, some problems need to be solved in order to come out with robust and reliable reader. In particular, several challenging design problems need to be solved. For example, most of the punched cards are old, over painted with spots of oils and have handwriting markers on the considered surface. These problems complicate the task of holes detection and recognition. In addition, several cards have been deformed due to physical damage or due to moisture available in the storing place. This deformation highly deviates the holes in the captured image from their reference centres. One more major challenge is that the card image does not contain discriminating visual features. The only visual information available is the pattern holes, while any thing else must be considered as noise.
The system presented in this paper is the first, in design and implementation, optical solution that is robust to all mentioned problems. Moreover, authors believe that this work is the first attempt to visually solve the problem of reading and retrieving the data from the Jacquard punched cards. It efficiently solves the considered problems like the over painting spots and correct the deformation of card.
The remaining of the paper is organized as follows. The next section presents background and basic information about punched cards, mechanical Jacquard, and electronics Jacquard. It also presents basic concept of visual technologies used in the system design like normalized cross correlation and 2d geometric image registration, particularly cubic inverse mapping. In Section 3, we present an overview of the proposed system along with the design challenges that make the problem hard. The detection process by searching the gradient space is presented in Section 4 while details of the image correction using 2D geometric inverse mapping is presented in Section 5. Description of building the real system and evaluation of its performance is presented in Section 6. Finally, conclusion remarks are in Section 7.

Punched cards and mechanical Jacquard
The automatic production of pattern weaving was made possible by the invention of the Jacquard loom, which is a mechanical loom invented by Joseph Marie Jacquard in 1801 [12,13]. The automatic Jacquard loom simplifies the process of textile manufacturing with complex weaving patterns such as brocade and damask. Jacquard loom enables mechanical production of the weaving patterns with the design stored in a series of punched cards. These punched cards are joined together to form a chain of connected punched cards. The punched card stores information using pattern holes which are most likely binary system. The presence or absence of pattern holes creates patterns in the fabric by controlling the way the yarns were raised or lowered [12]. The loom can handle enormously complicated designs. The loom is programmed by a chain of punched cards. See Figure 1(a) for samples of punched cards. Each card controls single throw of the shuttle. To produce a new pattern, the operator simply replaces one set of cards for another. Figure 1(b) is an example of a complicated mechanical Jacquard weaving. The original design produced by Jacquard was, in essence, to provide an effective means of communicating with the loom using the punched card [14]. The communication language was limited to just two words, "hole" or "no hole". It is most likely the same highly common binary system in today's modern computer. Figure 1(c) shows an example of mechanical Jacquard machine. There are many types of punched cards which are used in Jacquard machines. The most common types are VERDOL endless papers and VINCENZI cards. Only the latter type is considered in our design. The VINCENZI cards are available in three sizes as per the number of pattern holes which are organized over one, two, or three partitions. The larger one contains three partitions with 1320 pattern holes which is shown in Figure 2(a). The one with two partitions contains 880 pattern holes which is shown in Figure 2  Jacquard cards are fixed on a rotating carrier while being read by the Jacquard loom. Cylinder pegs and dedicated holes on the card are used for fixing purpose. These holes are called the peg-holes, they are greater in diameter than pattern holes. They can be clearly noticed in Figure 2(a,b).

Evolving to electronic Jacquard and design transfer
These mechanical machines, like the one shown in Figure 1(c), evolved to electronic Jacquard, like the one shown in Figure 1(d). The first electronic one was launched in 1983 by Bonas Machine Company Ltd. Due to recent advances in technology, the capacity of Jacquard machine is increased significantly. For example, a single-end warp control can be extended into more than 10,000 warp ends. This avoids the need for repeats and symmetrical designs and allows almost infinite versatility. The computercontrolled machines, shown in Figure 1(d), significantly reduce the down time associated with changing punched paper designs, thus allowing smaller batch sizes.
Many factories have huge number of cartons containing Jacquard Designs stored in punched cards. Designers and manufacturers are interested in transferring these old designs to new format suitable to be used with their new electronic machines. One reason for this interest is the difficulty and expense of redrawing identical designs to the original design. In fact, the new design has lower fine art values. The major challenge for redrawing these designs is that the Jacquard fabric designs are full of minor details such as type of weavings, Satin, Twill, Leno, Basket, etc. These fine details are usually invisible to human eyes. Figure 2(c) shows a sample of textile weaving. Because of these reasons it is very difficult to redraw old designs. Hence, a card reader is needed if we want to transfer these old designs to the new electronic Jacquard machines. Because of these reasons, a device that reads the punched cards and retrieves the data then writes these data in a format suitable to the new electronic Jacquard machines has to be made.
A French company, called ACTROM [15], developed a card reader using infrared transmitters and receivers located on opposite sides of punched cards, one transmitter-receiver couple for each hole's reference position. This requires 1320 infrared couple devices since the device is made in a way to be able to handle largest cards. It is expensive and difficult to build a system that controls such a huge number of couple devices. Even though this method is robust against handwriting and dirty spots, it has no efficient solution neither for small dirty objects that can cut off a beam of infrared, nor for deformation problem of card surfaces.

Normalized cross correlation
Normalized cross correlation function [16,17] is used to detect the peg-holes by matching its template against the image of punched card. The 2D map that represents the normalized cross correlation function R(t x , t y ) is computed using the following: Here w is the width of the template window, and h is the height of it. T(x, y) represents the template window, I(x + t x , y + t y ) represents the image shifted by t x and t y , The function slides through image, compares the overlapped patches of size w × h against template using the normalized cross correlation method and returns the comparison results.

Image registration
Image registration mainly aims at geometrically overlaying two images, the reference and the sensed images, of the same scene [17]. This alignment addresses the differences between two images. In our application, these differences are due to deformation of the image.
Image registration has a rich literature and has been extensively investigated by researchers [17]. This investigation has come out with large number of applications in industrial, medical, and services domains in addition to several surveys and tutorials. For example, works in [17][18][19] addressed the problem of image registration and presented a nice review of it. For recent works on image registration, one can be referred to [20,21].
Most of image registration methods consist of the following steps: • Feature detection: distinctive objects are detected manually or automatically by means of shape matching or other means. These objects have to be represented by marked points, control points, such as their centre of gravity. Cross correlation function between the sensed image and a template of the selected feature is an important tool for this purpose. • Feature matching: The correspondence between detected features of sensed and reference image is established in this step. • Estimation of transform model: In this step, parameters of mapping functions are estimated to establish the correspondence between sensed and reference images. • Image transformation: By this step, these mapping functions are applied to the sensed image in order to align it with reference image. Non-integer pixels coordination values are converted to integer values computed by a suitable interpolation method.

Design challenges
The problem of designing a reading system for old and used punched cards is challenging and difficult due to the following reasons: (1) The system is assumed to work in normal industrial environment where lightening conditions are not suitable for image analysis and vision computing. Changes in illumination intensity and distribution can be changed in uncontrollable manner. The colour intensity of the card's image pixels are drastically variables. (2) Cards are usually stored without any special care toward factors that may change the appearance like oil spots. Other dirty objects may leave colour change to the appearance when they touch the card. Such things may cause errors in readings; see Figure 3(a).  deformation must be carefully considered, otherwise it may lead to deviation of punched holes from their reference locations. Figure 3(c) shows a punched card where the region of deformation is marked by an ellipse. In this region, the hole centres are deviated from its reference position due to the deformation of the card surface.
Due to the mentioned reasons, pattern holes cannot be detected using classical intensity-based thresholding methods. A gradient-based search algorithm is developed to search for pattern holes after mapping the reference positions of pattern holes to the detected ones. To remove the deviation of the hole centres due to deformation, cubic 2D geometric transformation is used to map reference positions of the pattern holes to the detected image.

Algorithm and block diagram
The flow of the processes which start by reading the card image and end with writing the pattern code into the output file is depicted using the block diagram shown in Figure 4. The whole process consists of two main stages. First stage is the initial detection of pattern holes. This stage is preceded by substage to detect positions of peg-holes, then to create the reference map of pattern holes. This is detailed in Section 4.1. It is also worth noting that this first stage contains two substages: (1) detection of pattern holes by searching the gradient space of the intensity, (2) then refine the search to reject the false positives using the cross normalized correlation function. This initial detection stage along with the search in the gradient space and the refinement are detailed in Section 4.2. Second stage of the process includes 2D geometric mapping using cubic polynomial transformation. This stage aims at mapping the read image coordinates (u, v) into a reference map coordinates (x, y). This stage is proved to overcome problem of card deformation explained in the design challenges subsection. As preprocessing steps, the card is partitioned into smaller partitions to achieve more accurate mapping, and a set of control points is selected to achieve the computation of the mapping parameters. This mapping produces a new output image called the corrected image. As soon as this mapping is achieved, the processes of peg-holes detection, creating the reference map, and detection of pattern holes are applied once again on the corrected (new) image. As the block diagram shows in Figure 4, after the detection of the pattern holes, geometric mapping is performed if the detection is initial.
If this detection was applied on the corrected image and the geometric mapping is already performed, we tag the process as completed, and write the pattern code into the output file. The inverse geometric mapping including card partitions and control points selection are detailed in Section 5

Detection of pattern holes
The stage of pattern holes detection is presented in this section as well as the creation of the reference map. The reference map is used to initiate the search for the pattern holes via the gradient space. The creation of the reference map is presented first in the next subsection, and after that the pattern holes detection process is presented in the latter subsection.

Detection of peg-holes and creating the reference map
The card is mounted on rotating five faces prismatic carrier that carries the chain of punched cards. The card is engaged to the carrier using thin cylindrical pegs. Pegs interlock with large diameter holes on cards. These holes are called the peg-holes. Each partition has two peg-holes and 440 pattern holes organized in 28 columns and 16 rows. Detecting peg-holes centres allows us to create the reference map of the centres of pattern holes in the cards. In addition, reference map plays an important role in geometric transformation and producing the corrected image later. As shown in Figure 5(a,c), peg-holes can be easily detected and used to align the card image with the reference map.
The reference map of pattern holes is defined as the set of positions of all possible pattern holes on the card. The total number of pattern holes is either 1320 or 880 holes based on the type of the card. The holes of large type are organized in 16 rows and 84 columns. This yields 84 × 16 = 1344 holes but, as can be seen in Figure 6, there are four obsolete holes at each of the six pegs so the total number of holes is 1344 − 6 × 4 = 1320 holes. The pitch between every two holes is 4 mm. Taking this into account, all reference holes can be computed. These reference points will be represented in (16 × 84) two dimensions matrix, each element represents two coordinates of a position of a reference hole. Referring to Figure 6, it is found that number of holes between two pegs surrounding each partition of a card is 26 holes. Indeed, the horizontal pitch in (pixels) is   Vertical pitch is identical to horizontal pitch. In metric units every one of these pitches is 4 mm. Taking this fact into account, all (x, y) coordinates of reference holes can be easily computed.
To find out the centres of these holes, a template of peg is drawn as in Figure 5(b). The template has a white square mask and a black circle at its centre. The diameter of the black circle is the same diameter as the pegs' holes. A cross correlation function is used to achieve the detection of the peg-holes. The values of the cross correlation function is represented using two dimensions matrix whose dimensions are identical to card's image ones. Consequently, each element of this matrix gives the cross correlation value of a corresponding pixel in the image. A small region around each peg's reference position is inspected, the coordinates of maximum cross correlation value are the exact centre of a peg. Figure 5(c) shows result of this procedure where pegs are marked by crosses.

Detection of punched holes using reference map
After reference map is created, it is used to detect the pattern holes. To do this, the gradient of the card's image is computed and then used to detect whether there is a punch close to it or not. After that and in a refinement stage, the cross normalized correlation is computed to reject false positives. False positives here are the pattern holes which are detected due to the first three challenges mentioned in Section 3.1.

Detection using gradient
The gradient of the image intensity is computed starting at each reference position of pattern holes. A search for rising edge is performed in four main directions. As shown in Figure 7, the length of the search line in each direction is equal to half of the pitch between consequent reference positions.
Based on the number of detected rising edges, and the relative positions of the pattern hole centre (if it exists) and the reference centre, there are three cases: (1) The centre of the detected pattern hole is close to reference centre; see Figure 7(a). Searching the computed gradient of the intensity leads to four rising edges. (2) There is a considerable distance between the reference position and the centre of the detected pattern hole but less than its radius. Figure 7(b) shows that only three edges could be found in this case. (3) The distance between the reference position and the centre of the detected pattern hole is larger than its radius. Only one rising edge is detected as shown in Figure 7(c). This large deviation is due to the deformation in the card. Indeed, the image must be corrected using geometric transformation in order to solve this problem.  The pattern holes detected in the first case, where a rising edge at each of the four directions is detected, are highly reliable against disturbances such as spots of dirt and handwriting. Figure 8 shows the result of hole detection using image gradient where white small dots denote reference positions of pattern holes and large dots denotes the centres of the detected pattern holes.
If the reference position is punched then the computation of gradient will lead to four rising edges. Should these four rising edges be found, then this reference position is considered to match pattern hole. If less than four rising edges are found, then the reference position is considered solid and zero coordinates are written in its element inside the matrix of reference positions. By performing this operation, the matrix of reference holes will be modified into a new array that contains only the coordinates of detected pattern holes.

Refinement using normalized cross correlation
In order to reject false positives that appear due to image noise and appearance changes resulted from challenges mentioned in previous section, a refinement stage is suggested. This stage depends on computation of normalized cross correlation of the card image using a template of punched pattern hole. Searching the normalized cross correlation in the neighbourhood of each reference position helps to reject many false holes. Two observations must hold to confirm the existence of pattern hole in the neighbourhood of the reference position. One of them is the local extreme value nearby the reference position. The second is that the ratio of change of the normalized cross correlation value must be greater than a predefined threshold. The threshold is selected empirically but it is a constant value.
As can be seen in Figure 8, there are still some undetected pattern holes. These undetected holes happens when less than four rising edges result from the search via the gradient space. One may note that the refinement stage using normalized cross correlation does not help in detecting any missing holes, but helps in rejecting those holes that are wrongly detected. An elliptical marks region refers to errors where there are large drifts in location of reference position. These drifts lead to failure in detecting the holes.

2D geometric mapping for improved hole detection
We are going to discuss in this section the usage of image registration techniques to correct the image against deformation of the card's surface. In fact, the correction process, i.e., producing new corrected card image utilizes geometric transformation to generate a new corrected image. Before presenting the details of the geometric transformation, a preprocessing stage like selecting the control points after partitioning the image card into smaller regions is introduced.

Partition of card's image
It is hard to find single mapping function that models the deformation of the card image due to the nature of card usage. The pegs that are used to engage the card to the cylinder carrier change the nature of deformation between parts of image card so that there is no deformation function that represents the whole image. The image card must be partitioned into three sub images. As we explained previously two pegs delimit one partition of card. Narrow cards have two partitions and wide cards have three partitions. As can be seen in Figure 2(a) for wide card and in Figure 2(b) for narrow, the middle point between two inner peg-holes is the point through which vertical border line between right and left partitions passes.

Selection of control points
To achieve the estimation process of the parameters of the geometric transformation, a set of control points are selected. The control points here are a partial set of the correctly detected pattern holes with centre coordinates (u, v) along with there reference positions (x, y). Control points are preferable to be uniformly distributed over the considered sub-image. This is in order to make the deformation model accurate as much as possible. Each sub-image has 448 holes. We selected to divide the 448 holes into groups each of which has 4 holes as shown in Figure 9. The sub-image is scanned and one hole is taken from each group. If number of control points is less than 112, the case of empty groups, then scan operation is repeated and second hole is taken from groups that have more than one hole. The procedure is repeated until 112 holes are taken. It is worth noting here that it is common in the textile industry that every card has a minimum of 112 holes.
We need to find out a mathematical function that represents the deformation nature of each partition. This function is based on a selected set of the detected pattern holes. Centres of these holes are called control points. Computing drifts between real positions of these selected detected pattern holes and their reference positions allows us to find out a function that represents the deformation nature of each partition inside cards. Again, one may note that u,v are the coordinates of centres of detected pattern holes, in other words it is the distorted input, while x,y are the coordinates of reference positions of selected control points that are the corrected output.
One may note in Figure 9 that holes marked by × are chosen in first scan of holes. Holes marked by + are taken in the second scan.

Inverse mapping for geometric image rectification
After we have obtained the input/output data, i.e., the control points, the parameters of the geometric transformation model can be computed. The data we have are the centres of the detected pattern holes (u, v) and their reference positions (x, y). The aim of the geometric transformation is to generate a new corrected image with deformation being removed. In this new image, holes are re-arranged in new positions closer to their reference positions. A cubic polynomial inverse mapping function is used to model the geometric deformation. A method to compute the coefficients of this polynomial function is presented in the following.
Inverse mapping transformation maps each pixel in the output image (corrected image) into the pixel from the input image (distorted image). Figure 10(a) depicts the inverse mapping method. The bilinear interpolation method to estimate the grey level intensity values is depicted in Figure 10(b). The bilinear interpolation evaluates grey intensity of point (x, y) from the output image, given four closest pixels from the input image.
Inverse mapping function is formulated using cubic polynomial equation [16,18]  Using 112 control points, these two equations can be re-arranged and written in matrix compact form as follows: where A and B are two (10 × 1) vectors as follows: A = [a 0 a 1 a 2 a 3 a 4 a 5 a 6 a 7 a 8 a 9 ] T , while W is an array of (112 × 10) elements written as follows: · · · · · · · · · · · · · · · · · · 1 x 112 y 112 · · · x 3 112 y 3 112 ⎞ ⎟ ⎟ ⎠ . (5) Linear least squares solution produces the following values for A and B: Here W + = (W T W) −1 W T is pseudo inverse of the matrix W.
Using equations (2), all pixel coordinates of the corrected output image are mapped to the original image. Depending on most four neighbourhood pixels as in Figure 10(b), bilinear interpolation is used to compute grey intensities corresponding to each pixel according to the following formula [16,18]: where l = int(u), a = u−l, k = int(v), and b = v − k.
Here the function int() is the integer function.
As soon as the intensities of the new corrected image are computed, the detection process of the pattern holes is repeated as described in Section 4.2. Figure 11 shows one card image along with the detected pattern holes in both cases before correction (up) and after correction (down). The region with elliptical marker shows several undetected pattern holes before correction (upper image) which are detected again after correction (lower image).

Description of the real system
The presented system in this paper is designed, implemented, and built successfully to transfer the pattern designs from paper punched cards to a format that is acceptable by modern computerized weaving machines. The product is transferred to industry, and it is currently in site service for one of our industrial partners. Figure 12 shows external view of the system as final product ready to be transferred to industry.
The main components of the real system, as depicted in Figure 12, include a camera connected to computer and brushless DC (BLDC) motor along with its drive system to control the motion of the cards holder. The card holder is a rotary five faces prismatic object. At start up, a slot detector initializes horizontal position of cylinder. A capture command is sent to a webcam to acquire one image of the card. After the processing of the card image is completed, a rotation command is sent to BLDC drive to rotate the card holder one step. The rotation for one step brings new card to the field of view of the camera. Then a done message is sent to the computer to initiate a new capture command.

Controller design
The chain of connected punched cards are held on a rotary five faces prismatic as depicted in Figure 12. We used a BLDC Servo Motor BLDC to rotate the card holder by a step of 72 degree a time. The electronic drive is built using motion controller chip LM629 from National Semiconductor and TEXAS Instrument. This motion controller LM629 is controlled using an 8-bit Microcontroller ATmega162 from ATMEL. The said motion controller is in fact a complete solution dedicated to build closed loop servo motor system in position mode. The feedback signal is provided by incremental encoder.

Performance qualitative evaluation
To provide qualitative evaluation of the our proposed system's performance, we present results of a set of experiments that were carried out to test a group of 32 cards. A computer with 2 GHZ Intel Core 2 Duo Microprocessor and 2GB of RAM is used in the experiments. The number of correctly detected pattern holes is checked after two stages. The first one is after the initial detection using gradient search and normalized cross correlation while the second one is after applying the image correction using inverse mapping geometric transformation. Computations till the end of the initial detection takes in average 1.8 s. Total time of computations including the initial detection, image correction using inverse mapping, the detection using gradient space search, and normalized cross correlation for the second time (after image correction) take 2.5 s in average. Table 1 shows the numerical results of these experiments. First column (to the left) shows results from the initial detection using gradient search in four directions without image correction. The total number of actual pattern holes is 7309 holes, out of them only 7216 pattern holes are correctly detected while 93 of them are left undetected. This reports an error ratio of 1.27%. The average processing time is 1.8 s.
Second column (the right column) shows results after completing the processing stages. One may note that these stages are: (1) initial detection using gradient search and cross correlation, (2) image correction using inverse geometric mapping, and (3) the second round of detection using gradient and cross correlation applied on the corrected image. When processing is completed, only 3 pattern holes are left undetected out of the total 7309 pattern holes. This produces an error ratio of 0.041%. It is worth to mention here that we have found that one of these three pattern holes was already correctly detected before image correction, but the mapping function caused to miss it. The average processing time for the complete three detection processes is 2.5 s. Even though the proposed system is not a part of the manufacturing process, the time requirements of such industrial process are satisfied in our system.

Conclusions
In this paper, a new method has been developed to make a punched cards Jacquard reader based on image processing. The detection algorithm is based on normalized cross correlation to find out invariant features of punched card, i.e., the peg-holes of the card. Starting from these features, the reference positions of all holes are computed and stored in a reference map. The punched holes are discriminated from solid holes by computing gradient intensity in four directions, then false positives are rejected using cross correlation. Since the surface of several cards are deformed, this deformation needs to be corrected in order to return the deviated holes to their reference positions. A cubic polynomial inverse mapping function is used to modulate this geometric deformation. A method to compute the coefficients of this polynomial function is introduced. A brushless DC motor is used to control the motion of the chain of cards. This control brings new card to the camera field of view after completing the processing of previous card. The system is successfully designed and built. Evaluation tests show that the system produces high accuracy readings of the card pattern, with a satisfactory processing time as per the time requirements of the textile industry.

Disclosure statement
No potential conflict of interest was reported by the authors.