|
Received: June 2, 2000
Accepted: July 29, 2000
Ref:
Geradts Z. and Bijhold J. Overview of Pattern recognition and image processing in forensic science. Anil Aggrawal's Internet Journal of Forensic Medicine and Toxicology, 2000; Vol. 1, No. 2 (July-Dec 2000):
; Published: July 29, 2000, (Accessed:
Email Dr. Zeno Geradts by clicking here
: EMBASE Accession Number: 2004204947
 |
| Zeno Geradts |
|
-Zeno Geradts and Jurrien Bijhold |
In forensic laboratories, image processing and pattern recognition in image databases is given more attention for solving crime cases. In this paper a wide range of examples of forensic image processing and pattern recognition method is given.
In den gerichtlichen Labors, wird die Bildübersetzung und die Strukturerkennung in den Bilddatenbanken mehr Aufmerksamkeit für lösende Verbrechenfälle gegeben. In diesem Papier wird eine breite Strecke der Beispiele der gerichtlichen Bildübersetzung- und Strukturerkennungsmethode gegeben.
Dans les laboratoires légaux, le traitement d'image et l'identification de configuration dans des bases de données d'image est donné plus d'attention pour des cas de solution de crime. En cet article un éventail d'exemples de la méthode légale de traitement d'image et d'identification de configuration est indiqué.
The development of new techniques in image processing and pattern recognition has much influence on forensic science. By the number of camera surveillance systems that are installed it sometimes happens that a crime is recorded on videotape. If this occurs the tape can be used as evidence in court. The court often asks for image processing of the tape itself, to get a clearer image of the tape, and of comparison of the video images with a suspect. Furthermore the court sometimes asks if an image has been manipulated.
The second part of this paper covers pattern recognition in forensic image databases. Since there exist many kinds of image databases (where the fingerprints are the most well known), there is a need for searching on image features. The databases that we have investigated are: shoeprints, toolmarks, handwriting, cartridge cases and drugs pills.
In this paper a short overview will be given to the different techniques that are developed and used in the forensic laboratory.
Questions that are asked from our forensic imaging department by the court deal with :
Surveillance video images often have a poor quality. This is caused by light conditions, the resolution of the system and tape wear. In practice it is rarely that image processing (apart from contrast stretching) improves the quality with these kinds of images on a single image.
In the past [Bijhold 1997]1 we have investigated the different kinds of image restoration based on Wiener Restoration, Maximum Entropy and PDE's for still images. For artificial images this worked, however for real images it does not often work. For a sequence of images, the contents of these images can be combined in a final image, which is named the super resolution method.
 |
For the digitization of video images, we prefer to use uncompressed movies. This means that the data streams that should be handled are 30 MB/s. Furthermore there are also different options for digitizing. One can digitize 8 bits or 10 bits of information. The reason not to have compression is that the higher compression methods will often result in loss of quality. If this happens, the court often wants to know about the artifacts that can result from the compression method. There is even a substantial chance of the evidence getting rejected in the court. Since most often the time that needs to be analyzed, is a few minutes (robberies etc rarely take more time), we choose for uncompressed data streams. Furthermore a Time Base Corrector (TBC) has to be implemented in the system to compensate for stretches in the tape. If we choose to digitize 10 bits, the TBC should also handle 10 bits of information.
Furthermore we prefer to have source codes of the software that is used, and know exactly which image processing methods are implemented in the software. For this reason we have chosen for a Unix-based solution where we can use scripts, ImageMagick and Khoros software were the source code available. Often for processing a sequence of images, it is necessary to develop some own algorithms to analyze the processes. Furthermore with our own software, we can validate commercial packages, like Video Investigator of Cognitech.
In practice we see that often time-lapse tapes are submitted to the laboratory. The reason is that a normal videotape can handle just about a little more than two hours and if a continuous recording is used, one would have to use many tapes for a whole day recording. Obviously this makes the handling of the tapes inconvenient for the operators in shops and banks. By using a time lapse technique, often the tape can be made to handle 48 hours. However this means that less images per second are recorded than the normal 25 for PAL. For a tape of 48 hours, 1 frame per second is recorded.
Also the surveillance systems often record from more than one camera and the output is recorded on one tape. For this reason the following systems are often found at the crime scene [Geradts; 1999]2:
 |
There exist many kinds of noise in a surveillance system. If the noise is introduced by playing and digitization, it might be worthwhile to play the recording more often and average the final result. One danger with this method is however that the tape can be damaged if it is played more often.
Averaging over multiple frames can also reduce the noise. This can be valuable if there is a night recording with a camera and a scene that does not move. One problem arises when in certain frames moving people are seen (figure 2). Since these moving people are not present in most of the frames (that are averaged), obviously they wouldn't be visible in the final averaged frame. The value of this method depends on the part of the image that has to be visualized.
Super resolution is also a challenging field in forensic science. The idea is that if there is a small movement in the image, then a higher resolution image can be calculated based on a sequence of images. The camera or the scene itself should move to calculate such a super resolution image.
For validation of this method in court we tested it on Dutch license plates with known numbers. The way we calculated the super resolution image is as follows:
 |
 |
In figure 3 an example is shown with super resolution of a known license plate. As can be seen in the right hand photograph, the numbers are becoming clearer. The registration method is calculating the minimum of the standard deviation of the subsequent differences in the image.
This method is also implemented in commercial software, and other methods of super resolution are becoming available to forensic science.
However, sometimes it appears not to work, as is shown in figure 4. This was a compressed sequence, and due to image compression, artifacts in the super resolution image are introduced.
 |
 |
 |
In the past we have done some research on image restoration of a video sequence by calculating the movement or the blur of a lens system [Bijhold, 1997]1. These approaches most often do not work, since the blur function is not known. We have worked with different kinds of Wiener Restoration and Partial Differential Equations. In the examples that we created ourselves this did work. However in real casework we hardly ever can use these algorithms.
Other research that we are into is tracking a person in a video sequence [Geradts, 1999]2. This might be useful if the court asks for magnification of a sequence of images, since otherwise it takes too much time to work on the case manually.
For the comparison of a suspect with a video image there are several implementations.
One approach that we use is bringing the suspect to the same video recording system, and asking the suspect to stand in the same position as the people that were visible on the video recording. The problem with this approach is that the place should be available (often they have rebuild the place or the shop owners do not cooperate for emotional reasons) and that the suspect should cooperate, since it would be very difficult to get the suspect in the same position if he would not cooperate. Furthermore there should be other people who look similar to the suspect available, since we can draw a conclusion about the uniqueness of features that are combined.
 |
The court should use the output in an easy and convenient way for their interpretation of what happened at the scene of crime. Often we submit a written report with video prints to the court. However since they should watch the complete sequence, we sometimes send a composite video. The problem with this is however that the lawyer might ask if there has been any tampering with the digital evidence. So, we prefer to send a CD-ROM or DVD-ROM to the court with a hash-code based on SHA-1 of the movie-files that are on these discs.
 |
 |
Images are known for manipulation for a long time. Compared to other evidence (e.g. DNA or fingerprints), the question often arises if there has been any tampering with the images. For the analog videotapes we might investigate the signals, and we can say that the tape seems to be an original. If the tape has been tampered with a professional approach, we might not find out that it has been tampered.
One other problem that arises is with child pornography cases. In the Netherlands, the possession of virtual child pornography is not illegal. Virtual child pornography means that there is a composition of a child and an adult in such a way that the child seems to be abused, although it hasn't happened in reality. We had a case sometime back, where a lawyer claimed that his client's entire images that were uploaded on the Internet were virtual. The court asked us if we could investigate for manipulation.
We could prove that some of the images were manipulated by examining the edges and the light effects. Since all images were compressed by a lossy compression scheme, it was hard to find the fine details in the image. However it is hard to say that an image has not been tampered. This can only be claimed by testimonies and other ways of investigations.
Another question that arises is if a certain videotape has been recorded with a certain VCR that is used as evidence. Often there are signals available in the tape where this information might be checked. However it is a weak kind of evidence.
A stronger kind of evidence is available with cameras. If the question arises if a particular camera has recorded a video image or a still image, there might be information in the image itself. One of the strongest kinds of evidence was found with pixel defects in a camera. We have tested some cheap digital cameras and it appeared that there were enough pixel defects to identify the camera. In figure 6 an example is shown. Also cameras might hide serial numbers in files or by stenography. These ideas are under research at the moment. For this reason digital systems might have good methods for authentication.
Video image processing is important in the court. However often the expectations are too high. The responsibility of the forensic examiner is to inform the court on the possibilities, limitations and the artifacts that might arise from image processing. Many image processing methods should be validated before being used in the court, since the risk exists that if a certain method is not accepted in the court it might be very hard to use the method again in the court in other cases, whereas it might have been a method that could be useful in forensic investigations.
Pattern recognition in forensic science is often used in databases. This can be a database of DNA, fingerprints or paint. In this part we will focus on image databases of:
 |
In toolmarks examination it is possible to determine if a tool has made a toolmark based on striation marks or impression marks. Often the tool (e.g. a screwdriver) has characteristic damages and manufacturing marks. These characteristics may enable one to compare a test mark made with the tool with the toolmark found at the crime scene. And this can positively identify the tool. Often toolmarks are found at scenes of burglaries. For this reason the police have a database of toolmarks to link the various burglaries that have been committed by a suspect.
In this part of the research we handled striation marks. For comparing the striation mark we take a signature of the mark itself. This signature is sampled by taking fifty vertical lines and following the striation mark of a screwdriver itself with an algorithm.
In this way the user has to determine if the striation mark has been sampled correctly. Since a screwdriver can be positioned with different angles to the surface, at least five test striation marks should be made of each side of the tool in different angles. Since the user should compare all striation marks with the striation marks found at the crime scene, this work is very labor intensive. At least twenty comparisons have to be made for one screwdriver. The comparison with the complete database of all crimes that are committed, might take several days if it is done manually. A comparison algorithm is implemented that takes the fluctuations in marks, and allows partial zooming of ten percent. It appears that this algorithm works well for striation marks of good quality.
Furthermore tests are made with structured light [van Beest, 2000]5 to get a 3D-image of the striation mark. It appears that the shape of the tool itself can be digitized, and this might be easier for faster comparison, since otherwise all striation marks should be compared to each other, where also the shift and partial zooming has to be taken into account.
In the structured light approach, a sequence of different projection structures is created with a computer controlled micro-mirror projection. The whole procedure of projecting and grabbing the sequence of images takes a few seconds. The resulting images (object views) are processed and interpreted finally as one object view. In this resulting view the stripes carry the desired code numbers. Now the border lines in the image are defined and the X, Y and Z-co-ordinates of the object surface points calculated. The result can be viewed as a depth map (or image); it represents a 3D sampling of the surface. Accuracy and density of this method - known as Gray-Code-Projection depend on the largest number of stripes in the grids that could be handled by the camera system and by the accuracy of the geometric calibration. This information can be processed and visualized together with the 3D-data.
A small test was carried out for toolmarks in which five screwdrivers were used. Of these five screwdrivers, test marks were made with an angle of 45 degrees on a wax surface. The resulting striation marks were molded with grey silicon casting material. Then these marks were digitized with the structured light approach and with side light.
It appeared that there were some variations and artifacts in the image due to:
The way this algorithm works is by analyzing fifty lines of the image. One line contains the 512 pixels of depth information. The first line will be compared to the second, where the standard deviation of the difference of each pixel in a line is calculated. If the standard deviation is lower when a shift of one pixel is computed, then this shift is taken into account by the algorithm. Shifts of half a pixel are also possible with our algorithm. Then the averaged line will be compared to the third line, and also a shift of several pixels will be computed. A further improvement of this algorithm is also to allow shifts due to local variation in the striation mark. The averaged line results in the so-called "signature" of the striation mark. To validate the signature, the user can see the signature in comparison to the rest of the striation mark. Most optimal results can be obtained by manually selecting the area were the striation mark should be sampled.
The signatures can be compared with a database of signatures, just by calculating the standard deviation of the difference. Since there might be some shift, zoom and local variation in these, the algorithm also compensates in the same way as described above with the same algorithm.
It appeared that for the limited test set of six striation marks made with six different screw drivers that the algorithm was able to distinguish on :
Shoeprints are often found at the crime scene. It is hard for a suspect to hide shoeprints, since often they are latent in dust and can be visualized with various techniques. For shoeprints several databases might be useful in forensic investigation:
 |
In the Forensic Science Laboratory in the Netherlands a research study has been carried out for automated comparison algorithms of cartridge cases. This study is a part of an evaluation of the different systems, which exist on the market for databases of cartridge cases and bullets.
The reason for us to compare the different methods of image matching is that these methods are proprietary. For use in a forensic laboratory it is important for quality assurance to understand why a certain image is not found in top matching ranks and to have more background in the image-matching engine. Another reason for this research is to improve the results of image matching.
 |
When a firearm is loaded and fired, the mechanisms and various parts in the firearm that come into contact with the cartridge case cause impressions and striations that can be characteristic for the firearm being used. The striation marks on bullets are caused by the irregularities in the firearm barrel as well as larger and more distinct land and grooves of the rifling.
The ejected cartridge case shows marks (figure 9) that are caused by the firing pin and the breech face as the cartridge is repelled back in the breach by the force of rifling. The feeding, extraction and ejection mechanisms of the firearm will also leave characteristic marks.
In the Forensic Science Laboratory these marks on cartridge cases and bullets are compared with the test fired ones. Often the cartridge case is the most important forensic specimen in the identification of weapons, as bullets are commonly deformed by the impact. The examiner can also determine, using class characteristics what kinds of firearm or which make and model have been used.
This study handles different approaches for automated image matching algorithms on image databases of breach face and firing pin marks.
DRUGFIRE and IBIS are databases that can be used for acquiring, storing and analyzing images of bullets and cartridge cases. These two systems have been evaluated at our laboratory.
Both systems capture video images of bullet striations and of the markings left on cartridge cases. These images are used to produce an electronic signature that is stored in a database. The system then compares this signature to that of another fired bullet or cartridge case - or to an entire database of fired bullets and cartridge cases.
Both systems have image matching algorithms. The methods of image matching applied in these systems are not known. However patents applied by Forensic Technology describe state-of-the-art image matching methods. The system of IBIS is now used most often, and since the images are acquired in a reproducible way by a special kind of lighting, the ring light, this system will result in the best matching results.
For evaluation of these systems it is important that the user has a hit list (the order of the most relevant image matches that are displayed) that is reliable.
Other systems that have been described on the market are the system Fireball, the French system CIBLE and the Russian system TAIS. These systems also use image-matching techniques.
 |
For our evaluation of image matching algorithms we studied two kinds of images (Figure 10):
 |
In our research we assumed that there are no time constraints in the computation. So, this means that brute force methods can be used. The reason for this is that at first we try to use methods that are solid, and afterwards this method might be combined with faster approaches.
Since the conditions for digitization differ slightly, it is necessary to do some preprocessing of all images in the database. We did equalization and a normalization of the gray values of the images. After that we deleted the area outside of the primer, since this we would like to compare just the primer. Selecting a circle in the center of the image does this; the rest has a gray level 0 (figure 11)
For a computationally simple kind of comparison we take the variance of the difference in gray values between two images (which is also used in previous research [Geradts, 1995]8).
Sort this variance and the hit list is defined for the lowest variance. This approach is also possible with the image-matching coefficient, which is a better statistical approach. This will be looked at in the next section.
A classical technique for registering two images with translation misalignment involved calculating the 2D cross-image matching function [Anuta, 1970]9. Image registration algorithms are important for this kind of research, since the position of the marks is not known.
The maximum of this function yields the translation necessary to bring the images into alignment. This function has the disadvantage of being sensitive to rotation and scale change. Even small rotations of a few degrees can reduce the peak of the cross image matching function to the noise level.
By using the invariant image descriptors in place of the original images, it is possible to avoid this problem. One such descriptor is the log-polar transform of the Fourier magnitude, which removes the effect of translation, and uniform scaling into depended shifts in orthogonal directions [Casesent, 1976]10. The optimal rotation angle and scale factor can be determined by calculating the cross-image matching function of the log-polar transformed Fourier magnitudes of the two images. In our experiments we used 128 x 128 samples. This is adequate for the computational speed, whereas the details are still used.
There are extensive studies of image matching for tracking [Shi, 1994]11. Since tracking has many similarities with searching in a database of images, these kinds of methods can also be used for their speed. The features that are of interest are followed. Determining if a feature is of interest can be used for ranking it in a hit list.
Tracking also has the problem of registration, since the camera might move, with image intensities that change in a complex way.
One of the methods that appear to work for a wide range of tracking problems, and that works fast, is the Kanade Lucas Tomasi (KLT) method [Bres, 1999]12 . Good features are located by examining the minimum eigenvalue of each 2 by 2-gradient matrix. The features are tracked using a Newton-Raphson method of minimizing the difference between the two windows. Multiresolution tracking allows for even large displacements between images.
For comparison of images the number of points that could be tracked between two images is used as a measure for determining the top positions in the database.
We tested different image matching algorithms for marks on cartridge cases, using 19 matching pairs. For each pair several firearm examiners determined that they were shot with the same firearm.
In cases where the registration, and the light conditions among the marks in the cartridge cases was reproducible, a simple computation of the standard deviation of the subtracted gray levels put the matching images on top of the hit list. For images that were rotated and shifted, we have built a "brute force" way of image translation and rotation, and the minimum of the standard deviation of the difference is computed. For images that did not have the same light conditions and were rotated relative to each other, it was useful to use the third scale of the "à trous"-multiresolution computation.
It appeared that this method worked if the images of the firing pin marks are similar. The images of the firing pins are sensitive to variations in light condition. The "brute force" method takes much processing power and took 1 month to compute on a Pentium II-computer for 49 cartridge cases.
The ring light images were much more reproducible in image matching. The multiscale computation was not needed. All these images were correlated in the top position of the hit list with the brute force way of image matching and with the log polar measure. As the shapes of the region of interest are circular it is convenient for calculating in polar coordinates for computational reasons when using the difference method.
The log polar method works well in combination with the third scale and the ring light image. The computation time is seven days for a complete database of 4196 images.
A fast pre-selection method is carried out that is based on the KLT tracking equation. The details in the images that are prominent are selected in points. From each image these points are calculated one time and stored in the database. Then the position of those points is compared as a signature. In this way often the images that are most relevant are in the top positions.
For further improvement, it might be useful to have the refinement in which the user selects the areas that are relevant on the cartridge case for their marks. Sometimes the firearm examiner can have more information that some marks on the cartridge cases are damages that are not caused by the firearm.
The preprocessed images with the third scale à trous wavelet in combination with the log polar transform worked best and were fast for the database. To improve the speed of image matching the KLT-method could be used first for the top five percent for a faster pre-selection. The images and image matching methods that are used have marks that can be distinguished visually. For a large-scale evaluation it is necessary to modify the image matching method based on the information content of the mark.
With this method it is not possible to find a match if there are no visible breech face marks, however the ring light images might give results if the firing pin marks can be distinguished. Otherwise other marks on the cartridge case should be used.
The use of optical processors or parallel processors implemented in hardware is an option to improve the speed of the image matching.
 |
Many different approaches exist for searching image databases. For each kind of forensic image database, an appropriate method should be selected. Other databases that are under research are databases of faces, with the use of eigenfaces. The development of new algorithms that are faster, and implementation of search algorithms in hardware, might give new developments in these systems.
Also biometric systems will give more information for forensic science. An example of biometry is of course fingerprints, however there are many other kinds of research projects going on, e.g. gait, ear prints, retinas. For forensic investigation image databases can be important to validate the conclusions that are being drawn. An example of this is in the case of the uniqueness of ears and ear prints. In Dutch courts this kind of evidence is challenged, and there should be more research to have a more statistical background on uniqueness.
*Corresponding author and requests for clarifications and further details:
Zeno Geradts,
Netherlands Forensic Institute
Volmerlaan 17
2288 GD Rijswijk
Netherlands
N.B. It is essential to read this journal - and especially this paper as it contains several tables and high resolution graphics - under a screen resolution of 1600 x 1200 dpi or more, and preferably on a 17" or bigger monitor. If the resolution is less than this, you may see broken or overlapping tables/graphics, graphics overlying text or other anomalies. It is strongly advised to switch over to this resolution to read this journal - and especially this paper. These pages are viewed best in Netscape Navigator 4.7 and above.
-Anil Aggrawal
Click here to contact us.
This page has been constructed and maintained by Dr. Anil Aggrawal, Professor of Forensic Medicine, at the Maulana Azad Medical College, New Delhi-110002. You may want to give me the feedback to make this pages better. Please be kind enough to write your comments in the guestbook maintained above. These comments would help me make these pages better.
IMPORTANT NOTE: ALL PAPERS APPEARING IN THIS ONLINE JOURNAL ARE COPYRIGHTED BY "ANIL AGGRAWAL'S INTERNET JOURNAL OF FORENSIC MEDICINE AND TOXICOLOGY" AND MAY NOT BE REPOSTED, REPRINTED OR OTHERWISE USED IN ANY MANNER WITHOUT THE WRITTEN PERMISSION OF THE WEBMASTER
Questions or suggestions ? Please use ICQ 19727771 or email to dr_anil@hotmail.com
