Principles of Intelligent Automobiles

Xiubin Zhang  Muhammad Mansoor Khan

Principles of Intelligent Automobiles

Principles of Intelligent Automobiles

Xiubin Zhang Muhammad Mansoor Khan •

Principles of Intelligent Automobiles

123

Xiubin Zhang Shanghai Jiao Tong University Shanghai, China

Muhammad Mansoor Khan Shanghai Jiao Tong University Shanghai, China

ISBN 978-981-13-2483-3 ISBN 978-981-13-2484-0 https://doi.org/10.1007/978-981-13-2484-0

(eBook)

Jointly published with Shanghai Jiao Tong University Press, Shanghai, China The print edition is not for sale in China Mainland. Customers from China Mainland please order the print book from: Shanghai Jiao Tong University. Library of Congress Control Number: 2018954024 © Shanghai Jiao Tong University Press, Shanghai and Springer Nature Singapore Pte Ltd. 2019 This work is subject to copyright. All rights are reserved by the Publishers, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission or information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed. The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use. The publishers, the authors and the editors are safe to assume that the advice and information in this book are believed to be true and accurate at the date of publication. Neither the publishers nor the authors or the editors give a warranty, express or implied, with respect to the material contained herein or for any errors or omissions that may have been made. The publishers remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. This Springer imprint is published by the registered company Springer Nature Singapore Pte Ltd. The registered company address is: 152 Beach Road, #21-01/04 Gateway East, Singapore 189721, Singapore

Preface

Since the publication of the Chinese version of the book in 2011, it has been winning repeated praise from broad readers. The theory and technology of automobile intelligence provided by this book have been applied to the improvement of automobile technology and played a positive role in promoting the progress of automobile technology. The automobile industry has been developing for nearly 130 years. Since the first car appeared in the world, in order to get the most perfect car, people never stopped working on the improvement of automobile technology. Automotive Intelligent Technology (including “Intelligent driving vehicle” and “Driverless Vehicle”) is always the goal pursued by people. The so-called Intelligent Vehicle (IV) is the perfect mechanical and electrical combination of traditional automobile (including new energy power car) and modern sensing, information fusion, mathematical algorithms, microelectronics, computer, automation, artificial intelligence, and communication. Once the intelligent vehicle is integrated with the intelligent transportation system, the road capacity and traffic safety will be greatly improved, and the energy consumption of vehicles will be reduced. Over the past decades, I have led my Ph.D. and master students to carry out related research on automotive intelligent technology, and have obtained many invention patents. This monograph is based on these research results. The focus of the book is to explain the technical principle of intelligent vehicle from the depth of image information sensing, mathematical model, and algorithm base, so readers can understand and master the concept and design method of automobile intelligent technology fundamentally. It also enables readers to get many new ideas of science and technology and cultivate their innovative thinking ability. The book is written by Prof. Xiubin Zhang. Dr. Junhao Ying, Dr. Muhammad Mansoor Khan, Dr. Hao Wu, and Dr. Dongliang Lu have also undertaken some auxiliary work. The English version of this book has enriched some new technical

v

vi

Preface

contents in comparison with the Chinese version. Dr. Muhammad Mansoor Khan made serious checks and corrections to English Manuscripts. On this occasion, I would like to express my sincere gratitude to all the colleagues, students, and friends who have helped the publication of this book! Shanghai, China May 2018

Prof. Xiubin Zhang

Contents

1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.1 History of Automobile Technology . . . . . . . . . . . . . . . 1.2 Concept of Intelligent Vehicle . . . . . . . . . . . . . . . . . . . 1.3 Basic Content of Automobile Intelligent Technology . . 1.3.1 Overview of Existing Vehicle Safety Assurance Technologies . . . . . . . . . . . . . . . . . . . . . . . . . . 1.3.2 Development Trend of Automotive Intelligent Technology . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.3.3 The Basic Content of This Book . . . . . . . . . . .

. . . .

1 1 4 6

........

6

........ ........

12 13

. . . .

. . . .

. . . .

. . . .

. . . .

2 Vehicle Driving Safety Technology Based on IVP . . . . . . . . . . . . 2.1 The Basic Hardware Structure of IVP in Automobile . . . . . . . 2.1.1 System Hardware Configuration . . . . . . . . . . . . . . . . . 2.1.2 System Basic Working Process . . . . . . . . . . . . . . . . . . 2.2 Intelligent Identification Technique for Obstacles or Pits . . . . . 2.2.1 Implement Conditions . . . . . . . . . . . . . . . . . . . . . . . . 2.2.2 Outer Pole Constraint Principle . . . . . . . . . . . . . . . . . 2.2.3 Basic Algorithm Steps . . . . . . . . . . . . . . . . . . . . . . . . 2.3 Intelligent Detection and Control Technology of Safety Vehicle Distance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.3.1 Method for Rapid Identification of Vehicle in Front . . 2.3.2 Vehicle Distance Measurement . . . . . . . . . . . . . . . . . . 2.3.3 Driving Decision . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.4 Intelligent Technology for Preventing Vehicle from Impacting Pedestrians . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.4.1 Algorithm Principle . . . . . . . . . . . . . . . . . . . . . . . . . . 2.4.2 Threshold Extraction . . . . . . . . . . . . . . . . . . . . . . . . . 2.4.3 Similarity Measurement . . . . . . . . . . . . . . . . . . . . . . . 2.4.4 Advancement in Real Time Algorithm . . . . . . . . . . . .

. . . .

. . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

17 18 18 21 22 23 24 26

. . . .

. . . .

. . . .

32 33 49 52

. . . . .

. . . . .

. . . . .

62 63 65 66 67

vii

viii

Contents

2.5 Intelligent Rear View Technology of Automobile . . . . . . . . . 2.5.1 System Working Principle . . . . . . . . . . . . . . . . . . . . 2.5.2 Intelligent Rear View Kernel Algorithm of Vehicle . . 2.6 Automatic Identification Technique for Traffic Signs . . . . . . 2.6.1 Automatic Identification of Road Speed Limit Signs . 2.6.2 Automatic Recognition of Directional Signs . . . . . . . 2.7 Intelligent Technology to Prevent the Vehicle from Being Hit by the Rear . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.7.1 Device System Composition . . . . . . . . . . . . . . . . . . . 2.7.2 Identification of Trailing Vehicles and Active Control of Their Spacing . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . .

. . . . . .

. . . . . .

69 70 73 77 77 88

. . . . 102 . . . . 103 . . . . 103

3 Intelligent Prosecution of Dangerous Driving . . . . . . . . . . . . . . . 3.1 Intelligent Technology to Monitor Drunk Driving . . . . . . . . . . 3.1.1 Alcohol Sensing Theory . . . . . . . . . . . . . . . . . . . . . . . 3.1.2 Signal Processing Method for Alcohol Sensor . . . . . . . 3.2 Prevent Mistake of Stepping on Accelerator When Stopping Brakes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2.1 Anti Misoperation Device for Emergency Brake . . . . . 3.2.2 Anti Misoperation Algorithm for Emergency Braking . 3.3 Monitoring Whether the Driver’s Both Hands off the Steering Wheel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3.1 Sensing Structure and Pre-processing . . . . . . . . . . . . . 3.3.2 Sensing and Control Principle . . . . . . . . . . . . . . . . . . 3.4 Intelligent Recognition Technology for Driver’s Fatigue Driving . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.4.1 Hardware Configuration . . . . . . . . . . . . . . . . . . . . . . . 3.4.2 Core Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 Intelligent Monitoring Technology of Automobile Power and Transmission System . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.1 Intelligent Control of Automobile Engine and Transmission System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.1.1 Basic Functions of Power Control Systems . . . . . . . 4.1.2 Basic Structure of Power Control System . . . . . . . . 4.1.3 Three-Level Hierarchical Intelligent Control . . . . . . 4.2 Intelligent Identification Technology of Power and Transmission System . . . . . . . . . . . . . . . . . . . . . . . . . 4.2.1 Identification Device . . . . . . . . . . . . . . . . . . . . . . . 4.2.2 Intelligent Identification Algorithm of Abnormal Phenomena in Power and Transmission System . . . .

. . . . . .

. . . .

. . . .

. . . .

111 111 112 118

. . . 125 . . . 125 . . . 130 . . . 133 . . . 133 . . . 134 . . . 137 . . . 137 . . . 138

. . . . . 147 . . . .

. . . .

. . . .

. . . .

. . . .

147 148 149 150

. . . . . 155 . . . . . 156 . . . . . 158

Contents

5 Intelligent Vehicle Navigation and Traffic System . . . . . . . . . . . 5.1 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.2 Automobile Positioning Technology . . . . . . . . . . . . . . . . . . 5.2.1 GPS Positioning . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.2.2 Car GPS and GIS . . . . . . . . . . . . . . . . . . . . . . . . . . 5.3 Intelligent Traffic Route Selection Technology . . . . . . . . . . . 5.3.1 Basic Principles of Traffic Route Planning . . . . . . . . 5.3.2 Intelligent Route Selection Algorithm . . . . . . . . . . . . 5.4 Vehicle Communication Technology in Intelligent Transportation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.4.1 The Combined Action Principle of GPRS and GPS . . 5.4.2 The Role of 3G Technology in Intelligent Navigation 5.5 Real Time Increment Method of Navigation Map . . . . . . . . . 5.5.1 Information Incremental Updating Method . . . . . . . . 5.5.2 Application Example . . . . . . . . . . . . . . . . . . . . . . . .

ix

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

175 175 177 177 184 185 186 189

. . . . . .

. . . . . .

. . . . . .

. . . . . .

193 194 197 202 202 206

6 Vehicle Auxiliary Intelligent Technology . . . . . . . . . . . . . . . . . . . 6.1 Intelligent Control Technology of Car Light . . . . . . . . . . . . . . 6.1.1 System and Circuit Structure . . . . . . . . . . . . . . . . . . . 6.1.2 The Generation Principle of Controlling the Reference Voltage of Headlamp . . . . . . . . . . . . . . . . . . . . . . . . . 6.1.3 Intelligent Control of Anti Fog Lamp and Taillight . . . 6.2 Intelligent Switch Technology of Car Door . . . . . . . . . . . . . . 6.2.1 System Constitution . . . . . . . . . . . . . . . . . . . . . . . . . . 6.2.2 Kernel Algorithm of Intelligent Door Switch . . . . . . . . 6.3 Intelligent Control of Vehicle Windscreen Wiper . . . . . . . . . . 6.3.1 Rain-Sensing Principle . . . . . . . . . . . . . . . . . . . . . . . . 6.3.2 Practical Technology of Wiper Intelligent Control . . . . 6.4 Automatic Steering Technology of Automobile Headlamp . . . 6.4.1 Beam Distribution of an Automobile Headlamp at the Curved Road . . . . . . . . . . . . . . . . . . . . . . . . . . 6.4.2 Automatic Steering Technology of Automobile Headlamp . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.5 Automatic Response Technique for Parking Location . . . . . . . 6.5.1 Automatic Response Technique for Ground Parking Position . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.5.2 Location Response Technology of Underground Parking Lot . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . 211 . . . 211 . . . 211 . . . . . . . . .

. . . . . . . . .

. . . . . . . . .

217 220 221 222 227 235 235 243 247

. . . 248 . . . 248 . . . 253 . . . 253 . . . 255

7 Functions of Laser Radar in Intelligent Cars . . . . . . . . . . . . . . . . . . 263 7.1 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 263 7.2 Working Principle of Laser Imaging Radar . . . . . . . . . . . . . . . . . 264

x

Contents

7.3

7.4

7.5

7.6

7.2.1 System Structure of Laser Imaging Radar . . . . . 7.2.2 Mathematical Model of Laser Radar . . . . . . . . . 7.2.3 Target Characteristics and Laser Atmospheric Attenuation . . . . . . . . . . . . . . . . . . . . . . . . . . . Imaging Scanning of Laser Imaging Radar . . . . . . . . . . 7.3.1 Scanning Laser Imaging Radar . . . . . . . . . . . . . 7.3.2 Non-scanning Laser Imaging Radar . . . . . . . . . Common Vehicle Laser Radar . . . . . . . . . . . . . . . . . . . 7.4.1 Example of Laser Radar . . . . . . . . . . . . . . . . . . 7.4.2 Common Configuration of Unmanned Vehicles . Advantages and Disadvantages of Laser Radar . . . . . . . 7.5.1 Advantages of Laser Radar . . . . . . . . . . . . . . . 7.5.2 Disadvantages of Laser Radar . . . . . . . . . . . . . Perfect Configuration of Driverless Vehicle Sensors . . .

. . . . . . . . 264 . . . . . . . . 265 . . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

266 267 267 268 270 270 271 272 272 273 274

References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 275

About the Authors

Xiubin Zhang is Professor of Information and Control Engineering, and Professor of Electrical Engineering, School of Electronic Information and Electrical Engineering, Shanghai Jiao Tong University. His research interests include Information Processing and Intelligent Control, Theory and Technology of Image Information, Artificial Intelligence, Electrical Engineering, and Automation. e-mail: [email protected]

Muhammad Mansoor Khan is Associate Professor of Electrical Engineering, School of Electronic Information and Electrical Engineering, Shanghai Jiao Tong University. His research interests include Electrical Engineering and Automation, Information Processing, and Intelligent Control. e-mail: [email protected]

xi

Abstract

This book expounds the principle and design method of intelligent automobile based on the latest research results of intelligent automobile technology from the perspective of modern sensing and intelligent control. The contents include safe driving, intelligent monitoring for dangerous driving behavior, intelligent monitoring of vehicle power and transmission system, intelligent navigation and traffic intelligence system for vehicles, and auto-aided intelligent technology. This academic monograph is based on the achievements of the author and his graduate students devoted to the research of automotive intelligent technology for many years. The whole book expounds the principle of automotive intelligent technology in detail from information sensing, mathematical model, numerical algorithm, and so on. Therefore, it can facilitate readers to understand and master the design method of automotive intelligent technology, and further help students to broaden their academic perspective and cultivate their innovative thinking ability. This book can be used as a textbook for senior undergraduate and graduate students in automotive engineering, mechanical and electrical integration, modern sensing technology, detection and control, automation, computer, robot, and artificial intelligence. It can also be used as a reference book for professional technicians.

xiii

Chapter 1

Introduction

Since the advent of internal combustion engine-based vehicles at the end of the nineteenth century, there has been much research and development in the automobile industry. With basic industries become more developed and technologically advanced, today’s cars are becoming more refined. However, the automobile technology has not reached the level of perfection desirable compared to other technological advancements.

1.1 History of Automobile Technology More than a century ago, the first three-wheeled vehicles were running at a mere 18 km/h, but today’s supercar can accelerate only 3 s from 0 to 100 km/h. In the past 125 years, the pace of development in the automobile industry has been amazing, and the speed of the automobile technology is more amazing! However, the improvement of vehicle performance, like other modern technologies, requires a development process. From the long history of the development of the automobile industry, the history of the automobile technology development can be divided into three stages: the vehicle before the invention of steam engine, the steam car and the gasoline engine. Before the invention of the steam engine, man’s initial work was entirely dependent on biological means such as cows and horses. Then began to use natural forces, such as water powered waterwheel and wind-driven windmill, of course, the use of gravity in construction projects. This is the technical condition of the invention of the steam engine, not only at the stage of primitive, low productivity, and people mastering the knowledge dynamics of concepts and theories in physics theory development of the lower stage, as for conversion of thermal energy and mechanical energy of the fuel is lack of theoretical guidance and promotion.

© Shanghai Jiao Tong University Press, Shanghai and Springer Nature Singapore Pte Ltd. 2019 X. Zhang and M. M. Khan, Principles of Intelligent Automobiles, https://doi.org/10.1007/978-981-13-2484-0_1

1

2

1 Introduction

People, in the process of adapting to natural environment, the gradual accumulation of experience, and through the analysis of natural phenomena, to explore the laws of science, finally have the ability to grasp the laws of nature, taking the great pace of science, began to advance in a new era of scientific development. At this stage, the most prominent feature is that the power of the car comes from the steam engine. All technical improvements are centered on how to improve the efficiency of the steam engine and the reliability and stability of the mechanical structure. It can be said that this stage is a great progress in human science and technology, and also a period of a substantial leap forward in transportation. The steam cars were slow and heavy, and the people in the steam cars felt hot and dirty. To improve the engine, in 1838, British inventor Hennat invented the world’s first internal combustion engine ignition device, which has been called “a revolution in the history of world auto development”. In 1860, Etience Lenor produced an engine that was different from the fuel in the outer combustion engine, which allowed the fuel to burn inside the engine, known as the internal combustion engine. Since the latter half of the Nineteenth century, improvements and inventions around automotive technology have continued to emerge. At this stage, from the power plant to the structure design, from the appearance design of gas dynamics to the internal configuration, from vehicle speed to driving performance, from driving safety performance to ride comfort performance, vehicle technology has made considerable progress. During this period the most representative event is K. Benz’s first invention of gasoline vehicle and opened a new era of the automobile industry, so people are put in 1886 as the first year of the automobile, and K. Benz made the first wheeled car of the year (1885), as the car is born years. Subsequently, the improvement of automotive technology is still focused on automotive power plants to improve the efficiency and stability of vehicle engines. Since the Twentieth Century, the automobile technology has been improving day by day, and gradually moving from continental Europe to the United States. During this period, the improvements in vehicle technology were not limited to its power systems, and all kinds of new materials and technologies have also been used in automobile manufacturing. At the same time, a new type of transportation system and its management have begun to appear in the world. It is particularly worth mentioning, from the beginning of twentieth Century 60s, with the development of semiconductor technology, integration of electronic technology and automotive technology has become an inevitable trend of development, at the same time, electromechanical integration in automotive design has gradually formed the mainstream thoughts of a representative of the characteristics of the times. With the increase of vehicle population, traffic accidents caused by vehicles is increasing. Since the 1970s, all manufacturers have paid great attention to the research and development of automotive safety technology, including airbag, collision, and energy dissipation technology. At the same time, the technological achievements of computer science and technology have begun to have a significant impact on the development of global automotive technology.

1.1 History of Automobile Technology

3

Then, due to the limited resources of automobile fuel, human beings have to face a new choice-to explore new alternative energy sources and energy-saving technologies. All these technological advances have been carried out around oil saving, energy saving, and efficiency. Since the Twenty-first Century, all countries in the world have been actively organizing efforts to develop new fuel technologies for automobiles, including fuel cell vehicles, electric vehicles and hybrid vehicles. At the same time, automotive intelligent technology has also been highly valued. At present, more representative of the automotive intelligent technology products to count: (1) Since 1990, satellite navigation system and the technology of car driving assisted by laser, ultrasound, and camera have appeared. However, these are only the lowgrade smart cars whose functions and performance are not perfect and stable. (2) In the early 90s of the last century, Ford and Honda Corporation studied the application of neural networks and fuzzy logic systems in automotive dynamic control. Nissan also took the lead in the use of fuzzy controllers to control the speed regulation and anti-lock braking of the automobile transmission system. (3) In the middle of 1990s, Sakai and others considered how to develop a variable speed fuzzy controller under the condition of uphill or downhill driving. The fuzzy controller takes the real time speed and acceleration, throttle opening, road resistance, braking time and the current change as input, and gets the appropriate shift value after fuzzy calculation. (4) In 2006, Shanghai Jiao Tong University developed a prototype of driverless smart cars. It can be used as a short distance passenger vehicle in relatively closed areas such as university campuses, parks, airports and so on. (5) At present, GPS satellite positioning and electronic map selection technology have also been widely used in high-end cars. From the trend of automobile development, the realization of the intelligent vehicle is the inevitable result of the development of the world automotive industry, and also the pursuit of human progress in science and technology. Referring to the automotive intelligent technology, we can not do without the basic industry of the world and the foundation of the world auto industry. Only with the development of science and technology in the world, when the automobile industry has developed to a certain extent, can we realize the intelligent function of the vehicle on the basis of it. As far as the current world level of science and technology is concerned, some of the intelligent technology has been realized on the car, and some technology will be realized on the car. However, some technologies still remain conceptually, and it is necessary to rely on the development of scientific theory and the progress of basic industry to become a reality. If the car technology development process is compared to a Pyramid, the car technology that has developed over 130 years is similar to that shown in Fig. 1.1. Of course, the top of the pyramid has not arrived yet, and the development is endless.

4

1 Introduction

Fig. 1.1 Pyramid structure of automotive technology

1.2 Concept of Intelligent Vehicle Smart intelligent vehicle technology people might mistake with the possibility of traffic accidents. So, people hope the vehicle performance can reach perfection as soon as possible. The best car may, therefore, have a human-like visual, auditory, olfactory and tactile function. The reaction to stimuli is to be faster than human responses. In our surrounding, often have the rich imagination of students to showcase their imagination: smart car appearance is very small, but sitting inside but makes people feel very comfortable; the car can also be automatically driving, you just say where to go, the car will automatically find the best route, and then take you to reach the destination in; when driving at night, you will feel very tired, want to sleep, then you can get the car set on autopilot, then just say “rest”, the car will enter the “rest” driving state; not only open the door without a key, no remote control, as long as the owner says “open the door” the door will be opened, and so on. Without exception, people have had childhood fantasies. Although fantasy is very naive and romantic, fantasy often affects people to innovation. With the realization of innovation, step by step, the ideal draws closer to reality. Therefore, since the world has a car, people never stop on the road of fantasy, exploration and development. In other words, scientific discoveries and technological inventions are often inspired by fantasy and reverie at the beginning. Smart cars are no exception. For example, for the car to stop and hit a pedestrian or collide with another vehicle, people will think, if the car has a visual perception function, with pedestrians, vehicles or other objects, it can timely brake or slow down, this is how the ideal! For example, long-distance truck drivers continue to drive a few hours of freight cars, become

1.2 Concept of Intelligent Vehicle

5

exhausted there may be involuntary accidents. At such times, if there is a sensor in the cab, it can observe and identify the physiological state of the driver in real time. The accident can be avoided by preventing the driver fatigue driving through the control system. Fantasy and daydream always go beyond reality. When the reality progresses, the ideas that belong to fantasy and daydream will melt and merge with the new reality. It seems that fantasy and reverie always lead to reality, and reality always follows fantasy and reverie. But technology is the real thing, it is based on material. The intelligent car should be what kind of machine, it should have what kind of technical function. No one has yet been able to give it a very complete and accurate statement. Even in the technical world, there are many different opinions. For example, some people think that the smart car is a new high-tech car being developed, which does not require people to drive, and people can comfortably sit in the car and enjoy the drive. Because of this kind of car is equivalent to the “eyes” and “brain” and “foot” of the driver, computer and automatic control system like device, these devices have a complicated computer program, so this car can be like people will “think”, “judge” and “walk”, can start automatically, acceleration, braking, can automatically bypass obstacles on the ground. In complex circumstances, its “brain” can act according to circumstances, automatically select the best program, and then command the car to run normally and smoothly. Some people think that the smart car is a system composed of intelligent vehicles and intelligent highways. At present, the concept of the intelligent highway is not yet available, and it can be solved technically. Before the goal of smart vehicles, many intelligent technologies have already appeared, which have been widely used in automobiles. For example, intelligent wipers allow automatic sensing of rain and rainfall and automatically switch on and off. The automatic headlamp can be turned on automatically at dusk when the light is low. Intelligent air conditioning controls the volume and temperature of air conditioning by detecting the temperature of the human skin. Intelligent suspension, also known as active suspension, automatically controls the suspension stroke according to the road condition and reduces the jolt. To prevent drowsiness, monitor the blinks of the driver to make sure you’re tired, and when necessary, stop the alarm and so on. It gives a broad prospect for the intelligent vehicle. These statements are not without reason. People have made some incomplete summary of the smart car from their own experience, desire, and pursuit, which is ideal, and has been realized or can be realized. However, it is unavoidable to be biased. At the same time, the summary and summary of the intelligent automobile can never be comprehensive. Because science is advancing constantly, people’s thinking limitations will be constantly broken through. In terms of the current science and technology status, the so-called intelligent vehicle or vehicle intelligent technology should belong to a perfect transportation tool that is fully realized by using the existing technology. It has the functions of intelligent driving, automatic monitoring of vehicle driving condition and safe control of abnormal working conditions.

6

1 Introduction

However, it has to be said that no one in the world has yet given a scientific definition of an intelligent vehicle, and as far as the concept of intelligence is concerned, there is no exact definition so far. In other words, the smart car can not establish the clear technical indicators. That is to say, the intelligence of the so-called smart car must follow the pace of the world’s science and technology counterparts. But, to be sure, smart car is an entirely different concept from what they call an automatic car. The intelligent vehicle is a kind of intelligent machine which is developed on the basis of perfect technical function. It not only can automatically identify the road conditions and road traffic instructions, but also has the ability of intelligent identification and evaluation of the running and driving state of its own vehicles, and it can be perfectly integrated with the intelligent transportation. Of course, such a car must have near ideal safety, comfort and environmental protection. From a purely technical point of view, the smart car must have the characteristics of the intelligent robot, with visual and auditory sensory function, it also has a near perfect combination of sensors, computer and the movement of the machine system. Thus, the intelligent vehicle is a vehicle and a symbol of high-tech achievements. To put it simply, the so-called intelligent vehicle is a perfect logistics machine based on modern sensing, information fusion, mathematical algorithm, microelectronics, computer, automation, artificial intelligence and communication technologies fused together under the network environment.

1.3 Basic Content of Automobile Intelligent Technology It is not easy to summarize and classify the automobile’s existing intelligent technologies. It is more difficult to introduce the principle and technology of the smart car in an all-round way.

1.3.1 Overview of Existing Vehicle Safety Assurance Technologies In order to ensure the driving safety of the automobile, the high-end cars are equipped with airbags and collision energy dissipation devices, Anti-lock Brake System (ABS), Acceleration Slip Regulation (ASR) and Brake Assist System (BAS), Electronic Brake Assist (EBA), Cornering Brake Control (CBC), Dynamic Stability Control (DSC), Electronic Power Steering (EPS), Electronic Stability Program (ESP), Traction Control System (TCS), Tire Pressure Monitoring Device, Vehicle Stability Control (VSC) etc. (1) Once the vehicle collided with other hard objects, airbags can obviously play an important role in protecting the vehicle from the casualties, and the collision energy dissipation mechanism can play an effective role in counteracting the

1.3 Basic Content of Automobile Intelligent Technology

7

impact. On the current car’s airbag and collision energy dissipation technology, when the car collides with other vehicles, hard objects or pedestrians, it can reduce the casualties in the car and outside the car by ejecting the airbag inward and outward. However, this kind of technology, after all, is a “passive rescue” way. Once the car collides with other objects, the loss of the collision body and the vehicle is unavoidable, only the extent of the damage is reduced. “Collision energy dissipation” can only protect the personnel of the car, but it can not avoid the damage of the personnel outside the car. (2) ABS (Anti-lock Brake System) can be mounted on any vehicle with hydraulic brake. It uses a rubber air bag in the valve body to give the brake oil pressure. When you step on the brake, the air bag will fill the body of the ABS. At this point, the air bag uses the air barrier in the middle of the valve body to return the pressure, so to avoid wheel lock point. When the wheel is about to reach the next lock, the pressure of the brake oil can make the air bag repeat. For example, it can function 60–120 times in 1 s, which is equivalent to braking and relaxation, which is similar to the “point brake” of the machine. As a result, the ABS makes the wheel unlocked when it is on the brakes. It can avoid running out of control in the emergency brake and skidding on the wheel. It does not rub the tire with the ground at a point, thereby increasing the friction force and making the brake efficiency up to 90%. At the same time, it can also reduce the wear and tear of the brake disc, and can prolong the service life of the brake wheel drum, disc and tyre, reaching more than 200%. The test confirmed that the braking efficiency of vehicles with ABS in dry asphalt, rain and snow on the pavement can be respectively 80–90, 10–30, 15–20%. (3) ASR (Acceleration Slip Regulation) is an upgraded version of ABS, it can be equipped with the hydraulic expansion device, booster pump, hydraulic pressure cylinder, fourth wheel speed sensor on the ABS, with complex electronic systems and an electronic controller with acceleration system. In the slippage of the drive wheel ASR through the comparison of the wheel speed, the electronic system can determine the slippage of the drive wheel, and immediately reduce the carburetor intake and reduce the engine speed, thereby reducing the power output, to brake the slippage of the drive wheel. ASR is the most suitable power output device to reduce skidding and maintain the adhesion between the tire and the ground. In other words, the main purpose of ASR is to prevent car driving wheel slippage while accelerating, especially on the roads with less friction or covered with snow, hail, and frost road. Its functions include improving traction and maintaining the stability of the car. Driving on the slippery road, without ASR acceleration in driving wheel may cause, which makes vehicles prone to drift, especially the front drive vehicles prone direction control. Consequently, if the driving wheel slips, the entire vehicle will be slide to one side, which is extremely dangerous on the mountain road. In cars with ASR, these sort of incidents does not occur while they accelerate or decelerate, or at least can be minimized. In simple words, ASR enables the vehicle to steer along the correct path to avoid rollover.

8

1 Introduction

(4) The BAS (Brake Assist System) is the precursor of the EBA (Electronic Brake Assist) electronic brake assist. The utility model can judge the braking action of the driver, increase the braking force and shorten the braking distance when the emergency brake is started. According to the strength and speed of the driver stepping on the pedal, the braking force is increased in time, thereby providing an effective, reliable and safe brake. BAS for the elderly and women (lack of force) has a good effect to assist and the effectively shortens the braking distance. According to statistics, in the case of emergency, 90% of the drivers are not determined to step on the brake. The brake assistant system is especially designed for this situation. It can be determined from the detected driver’s brake pedal speed, when the driver quickly stepped on the brake pedal in an emergency, with insufficient stepping force and within 1 s time it will increase the braking force to a maximum value, hence shortens the braking distance under emergency braking. (5) EBA (Electronic Brake Assist) understands its braking behavior by trampling the speed of the brake pedal. If it detectable the “panic” increase of the brake pressure on the brake pedal, EBA will start all braking force in a few milliseconds, which is much faster than most drivers’ moving feet. EBA can significantly shorten the emergency braking distance and help prevent rear-end accidents. The EBA system relies on time base signals to monitor the motion of the brake pedal. EBA once detected on the brake pedal speed increased sharply, and the driver to continue to depress the brake pedal, it releases the stored 180 bar hydraulic to exert maximum braking force. When the driver releases the brake pedal, the EBA system turns to the standby mode. Since the EBA is able to exert the maximum braking force earlier, this kind of emergency brake assist can significantly shorten the braking distance. (6) CBC (Cornering Brake Control) is also called automatic curve control. When the vehicle is braking during the turning process, the CBC works with the antilock system (ABS) to reduce the risk of excessive cornering or understeer. Even under bad driving conditions, the stability of the car can be ensured. Some of the higher version of the ABS system already contains the CBC functionality. When the on-board check device detects that the car may be taxiing, the CBC system will reduce engine power and, if necessary, apply additional braking force to a particular wheel, thereby taking the necessary corrective actions for the vehicle. So, CBC can stabilize the car in 1 s. CBC contains complex computer control software, that is, “stability algorithm”. It can recognize the weight of trailers and automatically compensate for the increased vehicle load. (7) DSC (Dynamic Stability Control) is a technical extension of an accelerated anti skid system. It can ensure that the car has the best traction when cornering, to ensure the stability of driving. In order to make the car with good traction when cornering, DSC has the function of detection and control more advanced. For example, when detecting wheel speed, it can also detect the amplitude of the steering wheel, the speed of the car and the lateral acceleration of the vehicle, and judge whether there is any risk of skidding in the turning process according to the detected information. If there will be the danger of skidding or slipping, DSC brake hydraulic control system will immediately command the slipping

1.3 Basic Content of Automobile Intelligent Technology

9

of the wheels for proper braking, or to reduce the fuel quantity and the ignition delay to reduce the output power of the engine, the tire can prevent slipping under various driving conditions. It can make the vehicle in the starting, acceleration, turning process to get a good mark. In the car at high speed, it can also provide good handling, to prevent the vehicle swerves or drift phenomenon, which has precise control. (8) EPS (Electronic Power Steering) uses the power generated by the motor to assist the driver in the power steering. In different models, although the structure of EPS is different, its principle structure is similar. In general, it is made up of torque (steering) sensors, electronic control units, motors, decelerator, mechanical steering, and animal battery power. Its basic principle is that when the car turns, the torque sensor will “feel” to the torque of the steering wheel and the direction of the rotation. These signals are sent to the electronic control unit through the data bus to the electronic control unit (abbreviated as the ECU). The electronic control unit will send the action instruction to the motor controller according to the data signal such as the driving torque and the direction of the quasi-rotation. The motor will output the rotational torque of the corresponding size according to the specific needs, thus generating the power steering. If you do not turn, the system is not working and is in a “dormant” state. With the help of electric power steering, the driver feels a better sense of direction and more stable at high speed. Because the EPS does not work when the vehicle does not turn, the device has the characteristics of energy saving. At present, most of the high grade cars are equipped with EPS. (9) The ESP (Electronic Stability Program) is a traction control system. ESP system is composed of a control unit and a steering sensor (steering wheel angle detection), wheel sensor (detection of each wheel rotation speed), sideslip sensor (detection of body rotation around the vertical axis), a lateral acceleration sensor (centrifugal force detection when the vehicle turns) composed of. The control unit judges the running state of the vehicle according to the sensor signals, and then sends out a control command. ESP is particularly sensitive to excessive rotation or lack of rotation direction. For example, when the car turns to the left overly on the slippery ground (turning too fast), its tail will slide to the right. When the sensor senses the sliding of the vehicle, it will quickly brake the right front wheel to restore adhesion, resulting in an opposite torque to keep the car in the original lane. For the rear wheel drive car, when the steering is too much, the back wheel is often out of control and the tail slides. At this point, ESP will brake the corresponding front wheel, so it can make the body stable. When the steering is too little, in order to correct the tracking direction, ESP will brake rear wheel, thereby correcting the direction of travel. In other words, compared with other traction control systems, ESP not only controls the driving wheel, but also controls the moving wheel. The ESP system has included ABS and ASR, and is an extension of the two systems. Therefore, ESP is the most advanced form of automobile antiskid device. The difference between the two is that cars equipped with ABS or ASR can only

10

1 Introduction

react passively, while ESP powered cars can detect and analyze vehicle conditions and correct driving errors, and prevent them from happening. Of course, any function has a range of applications. If a motorist drives blindly, any safety device or technique is difficult to ensure his safety. (10) TCS is the traction control system. When the car is driving, the acceleration needs the driving force, and the side force is required in the turn. These two forces are all derived from the friction of the tire on the ground, but the tire has a maximum of friction on the ground. On a smooth surface with a small friction coefficient, the driving force and lateral force of the car are very small. The role of TCS is to make the car get the best traction in all kinds of driving conditions. The traction control system includes the vehicle embedded computer and the parameter detection sensor. The system can identify and determine the state of the body by detecting the speed of 4 wheels and the steering angle of the steering wheel. When the vehicle accelerates, if it detects that the driving wheel and the non driving wheel speed difference is too large, the system immediately determines the driving force is too large, so it will send out the instruction signal in real time, reduce the fuel supply of the engine and reduce the driving force, so as to reduce the rotation of the driving wheel. The system controls the steering intention of the driver through the angle sensor of the steering wheel, and then detects the speed difference between the left and right wheels according to the left and right wheel speed sensors, so as to judge whether the steering degree of the vehicle is consistent with the steering intention of the driver. If the steering is insufficient (or over steering), the system immediately determines the driving force of the driving wheel is too large, so the real-time instruction is issued to reduce the driving force so as to realize the steering intention of the driver. When the steering of the tire is moderate, the car can get the maximum driving force. In turn, if the tire can produce larger steering sliding, it will make the car have better acceleration ability. The system can use the steering wheel angle sensor to detect the driving state of the car, to judge whether the car is running straight or turning, and properly changes the slip rate of each tire. The mechanical structure of TCS can also prevent the idling of the driving wheel when driving on the wet and slippery road, such as snow, etc., so that the vehicle can start and accelerate smoothly, supporting the basic functions of vehicle driving. On a snowy or muddy road, TCS can ensure smooth acceleration. Besides, on the up and down steep slopes or dangerous rock roads, TCS can also properly control wheel skidding for the special off-road capability of the four-wheel drive vehicle. Compared with the traditional central differential lock device vehicle, the vehicle equipped with TCS has good driving and maneuverability that other devices can’t compare.

1.3 Basic Content of Automobile Intelligent Technology

11

(11) There are two ways to monitor tire pressure. (a) This is the use of the ready-made ABS sensor detection function to compare the number of revolving rings of the tire. If the pressure of the tire is insufficient, the circumference of the tire will become shorter. In four tires, as long as there is a tire pressure is insufficient, the number of revolving rings of the tire with insufficient pressure will be different from those of other tires, so that we can achieve the judgment of tire pressure. But this method of determination limits its detection range, only when the tire pressure is reduced to a certain value, it can be “perceived”. Moreover, when the pressure of the four-tire is as low as the tire pressure, this method can not “perceive” the tire pressure. (b) A tire pressure sensor is installed on the four wheels to directly sense the pressure of the tire. For example, the car produced by Corvette, BMW, Infiniti and other companies have been equipped with this type of tire pressure monitoring device. (12) The function of VSC is similar to that of DSC and ESP. VSC is developed from other technologies such as ABS and TCS. With the help of sensors and control units such as ABS and TCS, the VSC system continuously detects and processes the signals uploaded from the steering system, wheels and body sensors to determine whether the vehicle is skidding when it is bent. If skidding is found, the control unit has a trace brake on the wheel that needs to be a brake to help stabilize the vehicle’s running state. The system can further adjust the output power of the engine, that is to say, it can automatically control the braking and engine power of each wheel so that the body stability can be controlled without the need of driver intervention. In other words, as the VSC of a vehicle, it can control the sideslip caused by the drastic rotation of the steering wheel or the slippery surface of the road. Compared with other active security systems such as ABS, the VSC system has three major features. (1) Real time monitoring The VSC system can monitor driver’s manipulative action (steering, brake and throttle, etc.), road information and vehicle movement status in real time, and continuously send instructions to the engine and brake system. (2) Active intervention ABS and other security technologies mainly interfere with the driver’s action, but it can’t control the engine. The VSC system can actively control the throttle of the engine to regulate the engine speed, and adjust the driving force and braking force of each wheel to correct the excessive steering or steering insufficiency of the vehicle. (3) Beforehand reminding When the driver does not operate correctly or the road is abnormal, the VSC system uses warning lights to warn the driver.

12

1 Introduction

1.3.2 Development Trend of Automotive Intelligent Technology Although the safety technology of automobile body has reached a high level of perfection, and many control modes according to the multi-sensor signal, control instruction through the mathematical model of the operation and the output, but the technical level, still belongs to the simple and safe protection or control technology based on the model. For example, airbags, crash energy devices and ABS anti-lock braking systems only belong to simple safety protection techniques. ASR, BAS, EBA, CBC, DSC, EPS, ESP, TCS, tire pressure monitoring devices and VSC are model-based control techniques. It must be pointed out that it is not yet possible to know which of these systems contributes most to security. Through several simple tests, it is impossible to predict whether a car is better than the other in avoiding accidents. Therefore, the security of different vehicles should not be compared with the degree of participation of the stable control system. Similarly, traffic accident statistics can not prove that a stable control system of a manufacturer or a vehicle can reduce the accident rate more significantly than other vehicles. However, this conclusion has been proved that the stability control system can effectively reduce traffic accidents caused by vehicle out of control. The law of motion is still the determining factor in vehicle driving. Even in extreme environments, the stability control system does not absolutely prevent skidding, but can reduce the degree of sideslip. Even the more advanced technologies such as ESP, there are technical bottlenecks needed to break through. (1) In order to estimate the state variables of vehicle operation and calculate the corresponding motion control volume, the vehicle system needs a multi-level software structure with enough computing power. It can be said that the modern control theory based on the model is hard to adapt to the control of such a complex system. The designer must also seek for a robust nonlinear control algorithm. (2) Only when the perfect control function is interconnected with the engine and its transmission system, the control effect of the vehicle can be improved better. For example, the driving force of driving wheel can be estimated in real time by sensing technology, obtaining its mechanical transmission ratio, torque ratio and gear information from the automatic transmission. This can be used as a control feedback in order to significantly improve the starting comfort of a high-power car. It must also be pointed out that, under the constraint of the design concept, the body safety technology still belongs to “one-way safety system”. The so-called one-way safety system, that is, just consider the safety of drivers and riders, but not really based on the concept of safety in the road traffic system. The real sense of safe driving technology should and must consider both the safety of

1.3 Basic Content of Automobile Intelligent Technology

13

the vehicle itself and the safety factors of the road environment, including the safety factors of other vehicles and pedestrians. In a strict sense, this kind of technology does not yet belong to the intelligent technology of car safety driving. In the real sense, the intelligent automobile technology must have the sensing function of “seeing, listening and smelling”. And in the process of “seeing, listening, smelling” information, vehicle embedded system can rely on machine self-learning function, through a series of intelligent computing methods of pattern recognition, to achieve optimal control of vehicle running state.

1.3.3 The Basic Content of This Book This book tries to expound the auto intelligent technology from the functional classification point of view in the following aspects. 1. Intelligent Vehicle Driving Safety Technology Based on Image Information For the intelligent technology of automobile safety driving based on image information, this book calls it “automobile intelligent visual safety driving technology”. These include: intelligent recognition of obstacles and deep pits, intelligent inspection of safe vehicle distance, identification and collision avoidance of pedestrians and vehicles, intelligent rear view, intelligent recognition of traffic signs, etc. To a considerable extent, the traditional driving method depends on the mechanic manipulation of the driver. Usually the driver needs to control the direction, the throttle, the gear, the steering lamp, the light, the horn and so on. And the driver needs to keep his attention at all times to watch the road. When an emergency is encountered, the driver relies on his own skills and experience to deal with it. To a great extent, it depends on the subjective factors of human beings. For the same kind of emergency, different processing methods will have different treatment effects. Serious traffic accidents will occur when improper handling. So there is a lot of mental and physical pressure on the driver. Once the automobile has the intelligent technology of safe driving, people can make the safety performance of the car reach a close to perfect technical level. 2. Intelligent Control for Automobile Dangerous (or even illegal) Driving The most typical case of this is to drive a car after drinking or having been drunk. In order to solve this problem, the driver must be restricted by law, at the same time, the advanced technology of the car is also indispensable. This is the intelligent technology for monitoring drunk driving. Of course, intelligent inspection for dangerous driving also includes “Prevent the driver from mistakenly using the throttle as a brake when the brakes are needed”, “intelligent technology to monitor both hands away from the steering wheel during driving” and “intelligent technology for driver fatigue driving identification ”.

14

1 Introduction

3. Intelligent Technology of Automobile Power and Transmission System The composition of automobile power system mainly includes three parts: automobile engine system, transmission system and driving system. Among them, the engine system is the core of the car, it is the power source of the whole vehicle. The transmission system transfers the mechanical energy from the engine to the moving parts of the vehicle. The quality of the transmission system is related to the efficiency of energy transfer. The driving system is the end of the power output, the final executor of the effective power. Typical techniques in this field include: (1) Intelligent control of the engine The main task of intelligent control of the engine is that when the engine is limited by aerodynamics, thermodynamics and mechanical design, it can provide the fuel and air needed under various working conditions to ensure that the engine is in the best operation state. It includes fuel injection system, ignition timing control, idle speed control, exhaust gas recirculation, engine knock control and other related control, self-diagnostic system, backup system, etc. The intelligent control of engine also embodies the application effect of intelligent technology in automobile energy saving and environmental protection. (2) Intelligent control of the transmission system The main function of the transmission system is to realize the engine power transmission to the running system and other final execution mechanism, application of intelligent technology in the transmission system mainly lies in the power allocation problem to coordinate the vehicle, the vehicle better fuel economy and reliability. For example, the fuzzy controller is used to control the speed change of the automobile transmission system and the pressure modulation technique of the anti-lock braking system. At present, the intelligent control technology of automobile power transmission system is still being perfected. In view of the intelligent control theory and technology of automotive engines, it is necessary to rely on special literature to explain the theory and technology. Therefore, this book only focuses on some of the latest intelligent techniques and methods that are different from the general control of the power system. For example, the intelligent identification technology of abnormal state of power and transmission system. 4. Vehicle Intelligent Navigation and Traffic System In recent years, automobile navigation technology is a hot topic in the field of automotive intelligent technology. The automobile navigation system is one of the important contents of Intelligent Traffic System (ITS). It is a kind of real-time and efficient integrated management (control) system for transportation. Its main goal is to make full use of existing traffic resources to maximize the efficiency of existing traffic resources. The vehicle navigation system can provide the driver with a reasonable

1.3 Basic Content of Automobile Intelligent Technology

15

route information through the satellite navigation technology. It can detect the vehicle’s current location information through GPS or Beidou navigation system, and then determine a reasonable route for the driver according to the comparison between the electronic map and the target location. 5. Vehicle Auxiliary Intelligent Technology This aspect will focus on car light intelligent control, intelligent door switch, smart wipers, headlights automatic steering and parking location automatic response technology and so on. The highest level of smart car technology is the realization of unmanned vehicles. From a theoretical perspective, the driverless vehicle integrates information fusion, computer vision, integrated navigation, artificial intelligence, automatic control, map matching and architecture technology. It is the most advanced form of the application of intelligent technology. The so-called self-driving vehicle is actually equipped with a large number of sensors to collect all kinds of information about vehicle running, including location information, road information, weather information, obstacle information and surrounding moving object information. With the help of the expert system, the processor uses some information to calculate the next step motion parameters of the vehicle executing agencies, and then compares them with the current motion parameters, and finally forms control commands. The processor sends these control instructions to the controller of each actuator to adjust the motion parameters of the actuator to ensure correct operation of the vehicle. However, most of the current pilotless systems are basically in the experimental stage. This book will adopt the most realistic way to describe the technology of automobile intellectualization that has been or is about to be realized (industrialization) from the depth of the principle.

Chapter 2

Vehicle Driving Safety Technology Based on IVP

As the famous saying goes, “Human life is very precious, no one can deny the importance of life”, the car’s safe driving is the most important technical indicator that a car needs to achieve first. Car driving safety is very significant due to two aspects specifically, to ensure the safety of drivers and passengers in the car running, and to maximize the vehicle safety that it does not hit any pedestrians, vehicles and objects. This is one of the most important tasks that smart cars have been assigned specifically, an intelligent system for safe driving of automobiles. Therefore, the Intelligent Visual Perception of eyesight system is ensured, because intelligent driving is an ideal choice for car driving safety. The Intelligent Visual Perception (IVP), is a non-contact sensor technology, and this new scientific concept is based on the realization of the scene image understanding and understanding of methods and techniques. The so-called “view induction” is the whole process of the cognition and understanding of the external scene of the intelligent machine, which includes two parts: image sensing and view perception. Image sensing of machines is equivalent to human eyes and their retinas. It directly connects with the outside world through light, and makes the most direct and rapid response to the surrounding environment. The machine’s view perception is equivalent to the human brain center. It sends signals from retinal cells to the visual perception area of the brain to read and analyze light signals, so as to know objects’ distance, size, color, shape and other specific information. In simple terms, “view induction” can also be understood as the abbreviation of view perception. The “visual perception” combines the image information pickup and processing, recognition and understanding into a whole, and reflects the mechanism of the machine’s perception of the image information.

© Shanghai Jiao Tong University Press, Shanghai and Springer Nature Singapore Pte Ltd. 2019 X. Zhang and M. M. Khan, Principles of Intelligent Automobiles, https://doi.org/10.1007/978-981-13-2484-0_2

17

18

2 Vehicle Driving Safety Technology Based on IVP

Once the car has intelligent visual function, it is the same as the car has both eyes and mind. Such a vehicle can naturally replace the driver on the vehicle to observe and capture all kinds of information around it, so that it can enable the car to effectively avoid rear end collision, avoid obstacles and prevent pedestrians from accidentally injure.

2.1 The Basic Hardware Structure of IVP in Automobile Vehicle intelligent vision perception system includes two parts: hardware system and intelligent software algorithm. Among them, the hardware system is mainly composed of image sensors and on-board embedded systems (that is, on-board computer). Embedded system is called signal processor, as long as the vehicle embedded system configuration and memory clock frequency is high enough, and enough input/output interface and a suitable communication protocol, can bear the function of almost all automobile intelligent vision perception system.

2.1.1 System Hardware Configuration To observe the road conditions and its surrounding automatically, its essential that a car contains an “electric eye”, like the configuration of pinhole camera, as shown in Fig. 2.1. In Fig. 2.1, a total of eight pinhole cameras are required for the front, rear, left, and right of the car. Among them, 1 and 2 are the right and left pinhole camera set on the inner edge of the headlamp. 3, 4 are set to the right and left rearview mirror cover, near the edge of the outer edge, forward monitoring of the right and left pinhole camera. The 5 is pinhole camera arranged along the middle position of the windshield frame in front of the vehicle. 6, 7, respectively, set in the vehicle right, left rearview mirror, near the outer edge position, backward monitoring of the right and left pinhole camera. 8 is the rear view pinhole camera set on the trunk.

(a)

(b)

(c)

Fig. 2.1 Visual configuration of intelligent vehicle. a Front view, b Top view, c Back view

2.1 The Basic Hardware Structure of IVP in Automobile

19

The so-called pinhole camera is composed of CMOS (Complementary Metal Oxide Semiconductor) image sensor miniature camera. The photosensitive unit of CMOS image sensor and a storage unit is a photodiode, charge readout structure is a digital shift register, by controlling a group of multi switch, in order to charge each photosensitive element on the removed and sent to the public video output line (or video output bus). Its biggest feature lies in the following: (1) It has near ideal photoelectric sensing characteristics, high quantum efficiency, broad spectral response and low dark current. (2) Since its photosensitive surface is covered with a layer of transparent silica, the reflection loss is small and absorption loss is not present. (3) It is flexible in shape and size and can be made into an array of ring-shaped surfaces, so that it is easy to detect. (4) Its radiation resistance is much larger than that of CCD (Charge Coupled Device). (5) It can be made very small and easy to install. It almost doesn’t take up space. In addition to the eight pinhole cameras, the visual perception system also needs to configure the signal processor and controller. The principle diagram of the visual perception system composed of a camera (an image sensor), a signal processor, a controller, a voice player and a digital display is shown in Fig. 2.2. The auto configuration of pinhole camera (image sensor), these images are collected by the pinhole camera around the vehicle, signal transmission line further input these collected images to signal processor. Signal processor is the carrier of intelligent algorithm software. It has already solidified intelligent algorithm program, and it is the core hardware of vehicle intelligent vision technology. As shown in Fig. 2.3, the output is an analog of pinhole camera image signal, signal processor in IVP system consists of image input channel A1–A8, B1–B8 analog-todigital conversion module, image processing module, C control module D command output, voice command output module E, digital display and digital output module F commanded by the signal input module G. The intelligent algorithm program is solidified in the CDROM of the image processing program module C. In the eight pinhole cameras, cameras 1 and 2 are mainly used for collecting ground scene images, including road obstructions or deep pits, Lane driving signs, etc. Cameras 3, 4 and 5 are mainly used to collect speed limits, height limits and limit carrying signs , etc., while collecting traffic images on both sides of the body.

Pinhole camera

Controller Signal processor

Voice player Digital display

Fig. 2.2 Principle block diagram of vehicular IVP system

Control instruction output

20

2 Vehicle Driving Safety Technology Based on IVP To controller Image input

To voice player To digital display

Signal processor Other sensors digital signals

Fig. 2.3 Block diagram of signal processor. A1–A8-Image input interface, B1–B8-ADC module, C-Image processing program module, D-Control command output module, E-Voice command output module, F-Digital display instruction output module, G-Digital signal input interface

Cameras 6, 7 and 8 are used to collect images of the rear scenes of the car. All eight cameras acquire images parallel to the corresponding channel of the processor’s image input interface. Each image is initially transferred to an analog-to-digital conversion module through an image input channel and is converted into a digital image signal. The digital image signal obtained by processing and recognition of intelligent algorithm, signal processor to make accurate judgments on the characteristics of an object in the image, and generate a series of instructions through the control command output module D, a voice command output module E and digital output module F output display command. The controller generally includes the channel switch (electronic switch) and the digital to analog converter. The control instruction includes channel selection instruction and servo drive digital signal, and it can only contain channel selection instruction, and directly use 24 V DC voltage as control signal. When the control instruction contains the channel selection instruction and the servo drive digital signal, under the action of channel selection instruction, a specific channel switch is selected, and the servo drive digital signal can be transferred into the analog to digital converter along the gated channel, and is converted to analog drive voltage signal, which is used to drive the servo mechanism. When the control instruction contains only the information of the selected channel and the other values are zero, the system uses the existing 24 V DC voltage of the vehicle to drive the solenoid valve and the electric push rod directly. Under the drive of the controller, the visual system can accurately manipulate the normal driving of the car, and can prohibit the human error operation in real time. The voice player and the digital display are playing the signal processor’s decision results in real time under the action of the voice command and digital display instruction, so that the driver can grasp the driving information timely and adequately. The digital signal input module G is used to receive digital signals transmitted by the other sensors of the vehicle to the signal processor, such as speed, digital signals, etc.

2.1 The Basic Hardware Structure of IVP in Automobile

21

2.1.2 System Basic Working Process The intelligent algorithm program of the automobile visual perception system is more representative than that of the computer vision system, as shown in Fig. 2.4. Among them, image processing includes: image enhancement and filtering, image edge detection, image two-valued processing and image segmentation. The purpose of image enhancement is to improve the visual effects of images, and to facilitate manual or machine observation, analysis and processing of images. Image edge detection, image two-valued processing and image segmentation, etc., need to be determined according to the specific object and its technical purpose, whether or not relevant calculation is needed (all or part of the calculation). The key of IVP lies in how to realize the information extraction characteristics of the target image, it is to find out the characteristics of its information from the image, and can be identified according to the information characteristics of its properties, and then realize the target recognition. In Fig. 2.4, the so-called image feature extraction covers a wide range, and it has a great deal to do with the physical and morphological properties of the object being recognized, so there are several ways. Image features can be viewed globally or locally. The purpose of focusing on the local features of images is to reduce the computational complexity of the recognition process greatly. The basic idea of this recognition is especially applicable to the rapid computation of moving objects in moving vehicles. Feature extraction is a statistical pattern recognition method based on feature quantity. It mainly consists of two major steps: (1) extracting the characteristics of the representation pattern; (2) under the specific classification criteria, to determine the class to which the target object is to be identified. Feature recognition steps are shown in Fig. 2.4, which is basically the determination of the target category. The main program flow of image feature recognition and matching can be shown in Fig. 2.5. Among them, the classification method 1 is the pattern recognition classification method based on machine learning theory, and the classification method

Fig. 2.4 Intelligent algorithm program flow

Image acquisition

Image processing

Image feature extraction

Feature recognition

Control instruction output

To controller

22

2 Vehicle Driving Safety Technology Based on IVP

Scene to be recognized

Machine learning rule

Learning sample

Image acquisition

Learning sample feature space

Image preprocessing

Feature selection

Feature extraction

Template

Classification method 2

Common point matching

Classification method 1

Classification results

Three-dimensional reconstruction

Space point coordinate determination

Results instruction output

Fig. 2.5 Main program flow of image feature recognition

2 is the classification method based on the template matching similarity. Common point matching, 3D reconstruction and spatial point coordinate determination are the three dimensional scale detection steps of the recognized scene target. Only when the target object is identified, the stereo scale of the target object is measured, and finally the whole recognition process of the target object can be achieved. The following will be combined with the specific functions of automotive IVP, further describes the technical implementation of the method, in order to deepen the understanding of the process of identification shown in Fig. 2.5.

2.2 Intelligent Identification Technique for Obstacles or Pits The driver of the vehicle has a profound experience, because of heavy rain water or long-term overloading of vehicles, the road will be deep, as well as the vehicle accidentally dropped goods or vandalism are likely to increase the obstacles to the road accident. The road pit or obstacle is often the reason causing a major traffic accident of the vehicle, especially on the highway, the road is absolutely not allowed to have any obstacle, otherwise the consequences are unbearable to contemplate. How to make the vehicle having an intelligent detection and judgement function for the front, the road, the pit and the obstacle? Although there have been many

2.2 Intelligent Identification Technique for Obstacles or Pits

23

experts to carry out a lot of research work on this kind of technology, but in general discussion, the current research work is still basically stay in theory, can not enter the field of practical application. One of the more representative task includes the use of edge matching and scene reconstruction to extract targets in a traffic laboratory in a university in Europe, but only for short range targets. The multimedia R&D center of an Asian enterprise judges the existence of the target by affine transformation and region matching, but its application is limited by the region division. A university Robotics Institute in North America takes advantage of the anti perspective principle. Through the transformation and matching of several feature points to determine whether there are obstacles, but it needs to set up experience value and is not suitable for finding small targets. Some laboratories and institutes in China use the method of constructing lane and vehicle outline skeleton, and then determine target by skeleton search, but this is only applicable to identification of similar targets. Therefore, these methods are unable to be popularized because of their own technical limitations. The following is to provide the reader with a vehicle intelligent method for automatic identification of road pit and obstacles. This method allows the vehicle to automatically, accurately and rapidly identify the presence of a pit or obstacle on the road during the course of its journey. Once there are pits or obstacles, will be sent to the control mechanism according to the pit or the obstacle distance and braking deceleration control instruction, therefore, can effectively avoid the traffic accidents occurred.

2.2.1 Implement Conditions The signal processors in the onboard IVP system are used to process images taken from pinhole cameras 1 and 2 (see Fig. 2.1). While the pinhole camera was initially set up, a coordinate system was established on the car (see Fig. 2.6), and the internal and external parameters of the two cameras were calibrated. The camera internal and external parameter calibration is a basic work of IVP detection technology, because the geometric information of space object’s pose parameters by the camera imaging geometric model parameters and to determine the camera. Under the most conditions, these parameters must be obtained through

Fig. 2.6 Coordinate setting when identifying a pit or obstacle on a road

YW XW

ZW

24

2 Vehicle Driving Safety Technology Based on IVP

experiments and calculations. We call this process to calibrate camera parameters as calibration. Although the camera coordinate transformation, the distortion on the image correction process can simultaneously obtain the parameters of camera calibration, but the computer IVP system with different precision, the calibration parameters are not the same, so the use of the calibration method will be different. If higher measurement accuracy is needed, more complex imaging models are needed, and a high precision auxiliary calibration reference is required during the calibration process. In the virtual environment, the geometric model is only used to represent the basic structure of the object, but not to represent the details of the object surface, so in general, the calibration process of the camera should be simple and fast. For camera parameters calibration, there are many different calibration algorithms according to different system requirements. The calibration algorithm is divided into two categories, the traditional calibration algorithm and the self calibration algorithm, according to whether the calibration reference is needed in the calibration process (refer to Refs. [2, 3]). The dashed lines in Fig. 2.6 represent the respective field of view of each pinhole camera. As soon as the vehicle starts, the on-board IVP system is in the working state: the pinhole camera collects the image of the road scene in real time, and the left and right views are sent to the signal processor in parallel through the image channel. The algorithm software in the signal processor processes, identifies and understands the left and right views, and makes accurate decisions based on the results of the calculations. Once a pit or obstruction arises on the road, a control command is sent instantly to the actuator to automatically limit the speed or brake the vehicle.

2.2.2 Outer Pole Constraint Principle The key point of intelligent identification of road obstacles or pits is to calculate the 3D coordinates of spatial feature points. The calculation of 3D coordinates of spatial points based on binocular vision perception system, which is composed of two pinhole cameras, relies mainly on epipolar geometry constraint theory (referred to as outer pole constraints). The supposed epipolar geometric constraint, refers to the left (or right) of any point on the image, on the right (or left) the corresponding point on the image is only possible in a particular line, called the right polar line (or left polar line). With this constraint principle, it can greatly reduce the number of possible matching points. It puts one point on another image to find possible matching points, which reduces the original operation from two dimensions to one dimension, thus reducing computation and improving the speed of recognition operation. According to the above principle, the reconstruction of 3D images from two or more two-dimensional images usually uses the epipolar constraint principle to judge whether the projected points of the two images match or not. It can be said that epipolar constraint is the basic theoretical basis for finding the two corresponding points in the left and right views.

2.2 Intelligent Identification Technique for Obstacles or Pits

25

See Fig. 2.7, where the left and right cameras intersect, P is the point of the same three-dimensional scene observed at the same time from the left camera and the right camera, Ol and Or are respectively the left and right camera photocenter, l and r are respectively the normalized virtual imaging planes of the left and right cameras, the imaging points of P points on l and r are pl and pr respectively,   T T pl  xl yl 1 , pr  xr yr 1 . From the right and left the camera optical center respectively in the left and right normalized imaging plane points el and er are called left and right virtual imaging plane external poles. pl and pr are the normalized virtual imaging points at the space point P. They must be in the plane formed by the space point P, about perspective center point Ol and Or , which is called the outer polar plane. Outer polar plane and two straight lines from the left and right cameras, and two virtual imaging planes are called “left outer polar line” and “right outer polar line”. The left and right two outer polar lines of the common observation point P pass points pl and el and points pr and er respectively. When the left and right cameras of the binocular IVP system continue to observe the points on the line of Ol and P, such as P1 , P2 , P3 , Although the three imaging points P1 , P2 and P3 on the l are all one point pl , but the imaging points on the r are p1r , p2r and p3r , and the imaging points are on the right outer polar line, that is, on the line of er and pr and vice versa, that’s the outer polar line constraint principle. Similarly, when the P is in the location of the space changes, or that the binocular system observation point is changed, pl∗ and pr∗ imaging has new observation point P ∗ occurred in the normalized virtual imaging planes l and r , these will be respectively in the corresponding two outer polar lines. At this point, the imaging points on Ol P ∗ at l coincide with pl∗ , and the imaging points pri∗ (i  1, 2, 3, . . . , ∞) on r must be placed on the outer pole line er pr∗ . On the other hand, the imaging points at all points on Or P ∗ at r coincide with pr∗ , and the imaging points pli∗ (i  1, 2, 3, . . . , ∞) on r must be placed on the outer pole line el pl∗ . P* Left outer polar line

P

Right outer polar line

P1 P2

Πl

pl* pl

Ol Left outer polar point

P3

pr*

el

er Outer polar plane

Fig. 2.7 Outer polar geometry principle diagram

Baseline

pr

Πr

Or

Right outer polar point

26

2 Vehicle Driving Safety Technology Based on IVP

2.2.3 Basic Algorithm Steps The fundamental algorithm for automatic recognition of road obstacles and pits is implemented by the intelligent algorithm program in the image processing program module. Its detailed steps are as follows: 1. Collection of Road Scene Image Two pinhole cameras, the binocular image sensor system, real-time acquisition of vehicle front road scene image, and each collected image transmission along with the transport channel corresponding to the signal processor. 2. Eliminate Image Distortion The left and right views are removed to eliminate the distortion, respectively. In other words, from the pixel coordinates of each computer image coordinate system, the distortion point coordinates of pl and pr in the coordinate system of the corresponding normalized imaginary imaging plane image coordinates (xld , yld ) and (xrd , yrd ) are calculated. (xld , yld ) and (xrd , yrd ) are replaced by the mathematical model of image distortion correction, respectively. That is ⎧   ⎨ xld  1 + kl1 r 2 xlu l (2.1)   ⎩ yld  1 + kl1 r 2 ylu r ⎧   ⎨ xrd  1 + kr1 r 2 xru r (2.2)   ⎩ yrd  1 + kr1 r 2 yru r Through the inversion operation, the ideal coordinates (xlu , ylu ) and (xru , yru ) of the normalized virtual imaging plane coordinate system are obtained after rectification. 2 2 2 2 + ylu ; in the formula (2.2), rr2  xru + yru ; Among them, in the formula (2.1), rl2  xlu kl1 and kr1 are low order radial distortion coefficients in the left and right virtual imaging planes, respectively. Then, the ideal point coordinates (xlu , ylu ) and (xru , yru ) are used to replace (xld , yld ) and (xrd , yrd ) respectively, and the equation is substituted, ⎧

⎪ il  xld dxl + clx ⎪ ⎪ ⎨

jl  yld dyl + cly (2.3) ⎪ ⎪

⎪ ⎩ slx  dyl dxl ⎧

⎪ ir  xrd dxr + crx ⎪ ⎪ ⎨

jr  yrd dyr + cry (2.4) ⎪ ⎪

⎪ ⎩ srx  dyr dxr

2.2 Intelligent Identification Technique for Obstacles or Pits

27

Find the new coordinate value for pl and pr , that is, the new pixel position. On the left and right camera image coordinates, all pixels coordinates (il , jl ) and (ir , jr ) pass through the calculation, we can obtain the pixels on the screen in the ideal arrangement, namely real scene image restoration, or, to obtain the ideal image of left and right view, they can reflect the real scene. In type (2.3) and (2.4), dx and dy are the distances between the unit pixels in the x and y directions in the virtual imaging plane, and the sx is the ratio of the pixel diameter of the y and x directions, i.e. aspect ratio; cx and cy are the pixels coordinates of the camera’s optical center in computer imaging plane; In formulas (2.1) and (2.4), the foot labels l and r represent the left and right cameras (the same below). So-called normalized virtual imaging plane, which excludes the physical concept of optical imaging, builds the imaging plane in a virtual position at the focal length of the lens located in front of the lens. Compared with the pinhole camera physical imaging plane, it possesses several advantages as follows: (1) The actual distance units instead of pixels, because of the different visual sense of image sensor, the focal length of the lens and the image resolution and other parameters caused by the calibration and calculation difficulty in the more visual sense system. (2) As the optical axis passes through the origin of the coordinate system, there is no deviation of the optical axis and the origin of the physical imaging plane. (3) As a result of normalization, it facilitates the operation of matrix system and assumes the intermediary role of transformation. (4) The coordinates of the imaging points on the normalized virtual imaging plane in the camera coordinate system are the homogeneous coordinates of the twodimensional coordinates. (5) If the imaging plane is built on the imaginary plane before the lens is not present, it can solve the problem that the mathematical model is not intuitionistic. As a result, the accuracy of the discussion is ensured while the discussion of the problem is simplified. Figure 2.8 shows left and right two views of obstacles occurring in front of the vehicle, which are collected by the on-board 1 and 2 pinhole cameras and are removed by distortion. 3. Finding and Matching Common Corners of Obstructions or Pits To confirm whether there is an obstacle or a pit in front of the vehicle, it is necessary to look for the obstacle or pit, and match the common corners points from the left and right view. For this reason, the view window is first intercepted, as shown in Fig. 2.8. The size of the window is expanded along the top, bottom, left, right four directions of the camera axis of the pinhole camera, and a certain number of pixels are formed. Among them, the transverse (column pixels) is used to monitor the right column pixel carriageway width number N is appropriate; longitudinal (line pixels) is used to identify the operation cycle to the maximum allowed number of pixels for M

28

2 Vehicle Driving Safety Technology Based on IVP

(a)

(b)

Fig. 2.8 Road obstacle image acquired by vehicle in real time. a Left view of road ahead of vehicle to remove distortion. b Right view of road ahead of vehicle to remove distortion

for the upper limit, so formed is the same as the size and pixel coordinates of their corresponding to each other about two view window with M × N . In the intercepted window, the corner detection algorithm based on image gray is adopted, and several corner points are represented by calculating curvature and gradient to represent the most characteristic edge points. Then, in order to find the corner points, under the guidance of epipolar constraint principle, we find the matching corners in the left and right views. Take the road obstacle shown in Fig. 2.8 for example, the highest point b of the stone closest to the vehicle is on the outer pole line Ll and Lr in the left and right view (see Fig. 2.9). The point b in Fig. 2.9 is one of the common corners of the left and right two views that are automatically searched through the epipolar constraint. 4. Determination of 3D Coordinate Values of Object Points Determine the coordinate system involved in the 3D coordinates of the object points and their relations, namely, the pinhole camera geometry model, as shown in Fig. 2.10. In the diagram, the world coordinate system OW XW YW ZW needs to determine the direction and origin of the coordinate axis according to the specific environment, and (XW , YW , ZW ) represents the 3D coordinates of the object point P in the world coordinate system. In the camera coordinate system oc xc yc zc , where oc

(a)

Ll

b

(b)

Lr

ZW

YW Fig. 2.9 Example of epipolar constraint of binocular view

XW

2.2 Intelligent Identification Technique for Obstacles or Pits

29

zc

pij

i

j

xc yc

oij

P ( X W , YW , ZW )

pu pd x

o

oc

p ( xc , yc , zc )

y ZW OW

XW YW

Fig. 2.10 Geometric model of pinhole camera

is the origin point, which is defined in the optical center of the camera lens, xc , yc axis in the lens plane, and they are vertical to each other. The zc axis coincides with the optical axis, and (xc , yc , zc ) represents the 3D coordinates of the object point P in the camera coordinate system. The computer image coordinates oij ij, the origin point oij is located in the lower right corner of the CCD image plane, zc  −f plane where the actual image in camera coordinate system, f is the effective focal length of the camera, i and j are respectively the number of columns and rows of pixels, the unit is pixel. The i and j axes are parallel to the x and y axis, and they are in the opposite direction, and the pinhole imaging “upside down” phenomenon is the same, when representatives of cameras in the output of the CCD signal, the signal for the horizontal and vertical flip, pij is a projection imaging point in computer image coordinates oij ij on a the space point. In the virtual image plane normalized coordinates oxy, where origin point o is defined in the intersection of the optical axis of the camera and the unit focal plane zc  1, pu (xu , yu ) represents the ideal imaging point coordinate in normalized virtual imaging plane on P, and pd (xd , yd ) is a coordinate actual imaging point deviation of pu caused by lens radial distortion. T  In the left and right views, the matched corners can be expressed as pˆ l  il jl T  and pˆ r  ir jr in the computer image coordinate system oij ij, and by the mapping relation of world coordinate system: ⎤ ⎡ ⎡ ⎤ ⎡ ⎤ XW ⎤⎡ f 0 c r r t r i x x 1 2 3 x ⎢ ⎥ 1⎢ ⎢ ⎥ ⎥⎢ YW ⎥ ⎥⎢ (2.5) ⎥ ⎣ j ⎦  ⎣ 0 fy cy ⎦⎣ r4 r5 r6 ty ⎦⎢ ⎣ ZW ⎦ zc r r r t 1 7 8 9 z 0 0 1 1 Or

30

2 Vehicle Driving Safety Technology Based on IVP

      1 1  pˆ P P  M  K Rt zc zc 1 1 1

(2.6)

T  The process of finding the three-dimensional coordinates P  XW YW ZW of a point in a space object is as follows: T  (1) Substituting the left side of formulas (2.5) or (2.6) by pˆ l  il jl , then T  corresponding space point coordinate PW l  XW l YW l ZW l of the left view can be obtained. T  (2) Substituting the right side of formulas (2.5) or (2.6) by pˆ r  ir jr , the T  corresponding space point coordinate PW r  XW r YW r ZW r of the right view can be obtained. T  (3) Finally, the coordinate of the space point is obtained PW  XW YW ZW , and PW  21 (PW l + PW r ). In formulas (2.5) and (2.6), zc is a non-zero scaling factor that guarantees normalization⎡of both homogeneous coordinates and sometimes also the letter s. The matrix ⎤ r1 r 2 r 3 R  ⎣ r4 r5 r6 ⎦ describes the rotation relation between two coordinate systems, r7 r8 r9 whose row and column vectors are unit orthogonal vectors. R is an orthogonal matrix, T  so there are only 3° of freedom. Column vectors t  tx ty tz , which describe the translation relations between coordinate systems, are called translation vectors, with 3° of freedom. The 6th parameter, consisting of 3 rotation parameters  and 3 translation parameters, is called an external “parameter”. The matrix R t with 3 × 4 is called the ⎤ ⎡ fx 0 cx ⎥ ⎢ outer parameter matrix. Where cx and cy of K  ⎣ 0 fy cy ⎦ are pixel coordinates 0 0 1 of the photocenter o in computer imaging plane, fx and fy are called logical focal lengths, and their values are related to the actual focal length f . Although the optical imaging system of camera has only a physical focus, but in the image may have two different logical focal length, fx and fy also known as scale factors of the axis i and j in the computer image coordinates. The matrix K with 3 × 3 is called the inner parameter matrix, in which the elements fx , fy , cx and cy are called the intrinsic parameters of the camera.   In formula (2.6), the matrix M  K R t with 3 × 4 completes the projection T  process from the three-dimensional coordinate PW  XW YW ZW to the two-

2.2 Intelligent Identification Technique for Obstacles or Pits

31

 T dimensional coordinate p  i j , that is the geometric projection of 3 ⇒ 2 , so M is called the projection matrix. It can be further seen from the formula (2.5) that the projection matrix is multiplied by the inner parameter matrix by 4° of freedom and the outer parameter matrix with 6° of freedom, which results in 10° of freedom. 5. The Identification of Pit or Obstacle Establish two decision thresholds, including the negative threshold eh of height and the positive threshold ec of pit. The eh indicates the allowable roughness of the road that the vehicle can allow, where the ec indicates the uneven depth of the roadway which vehicle can allow. Such that assuming origin of point OW in the world coordinate system which would be OW XW YW ZW is arranged on the wheel and the ground contact points, considered vehicle eh  −0.1 m can allow the pavement roughness height would be 10 cm, also assuming that vehicle ec  0.05 m can allow uneven pavement pit depth equal to 5 cm. In other words, when the height coordinate is T  XW ≤ eh in the space point P  XW YW ZW gets detected in the pavement, which shows that road has obstacles, then vehicle would bypass or brake, otherwise, the obstacles on the road is likely to cause damage to the vehicle chassis, or even major traffic accidents is plausible. When the height of the coordinate is XW ≥ ec T  in the space point P  XW YW ZW gets detected in the pavement, which shows that road has pits, then vehicle bypasses or brakes, otherwise, the pavement pit is likely to cause vehicle’s axle or chassis damaged, or serious traffic accidents. A typical example is shown in Fig. 2.11. As shown in the diagram, there is shear settlement besides the collapse of the road surface. Among them, a, b and c are common feature points of left and right views, which are found by steps 3 and 4 above. After calculation, XW (a)  8 cm > ec is obtained, which shows that this is obviously a “deep pit” that must be avoided. Meanwhile XW (b)  −11 cm < eh , and |XW (b) − XW (c)|  8 cm, which shows a “road pit” that must be avoided.

b

b

c a

Fig. 2.11 Examples of pavement collapse and settlement

a

c

32

2 Vehicle Driving Safety Technology Based on IVP

6. Vehicle Driving Control When XW ≥ ec or XW ≤ eh , it has been shown that the space point P  T  is a pit or obstacle point. At the same time, the signal processor XW YW ZW can make the control decision accurately according to the coordinates of the point ZW , speed and results in the form of actions to control speed mechanism and the voice prompt. Control command, warns the driver to pay attention on road condition ahead, and vehicle is slow down automatically by the speed control mechanism and can be stopped before any pit or obstacle. Let suppose ZW  100 m and vehicle speed is Vt  80 km/h at this point, vehicle must slow down with a negative acceleration of a  −2.47 m/s2 to ensure that the vehicle would stops within 9 s just before the pit or obstacle encountered on the road as shown in Fig. 2.11, the intelligent technology is responsible to stop the vehicle. Repeating the cycle process of the image acquisition, recognition, operation and decision control, the vehicle can recognize and judge the pit or obstacle of the road in real time and continuously.

2.3 Intelligent Detection and Control Technology of Safety Vehicle Distance Keeping a certain distance between front and rear of vehicles is a problem that must be paid attention when driving safely, especially on the motorway. But how can we maintain safe distance between vehicles? If the driver’s vehicle distance is controlled solely by the driver, there are clearly fatal weaknesses. That are (1) Relying on visual observation makes difficult to maintain the exact distance. (2) For a common driver, it is impossible to drive the vehicle according to the prescribed speed accurately, due to “mental arithmetic” and other reasons it’s difficult to maintain proper distance with previous car. (3) Influenced by driving experience and driving psychology, it is possible to neglect the safety distance and get out of control. In order to resolve this issue, ultrasonic, millimeter wave radar and laser technology have been applied to vehicle system, so vehicle’s detection distance performance would improve. As far as the prior art is concerned, the ultrasonic measuring distance is short, and the millimeter wave radar has great influence on the ranging under the interference of the electromagnetic wave. And the point of laser measurement is little, and the imaging laser technology (structure light imaging technology) is too complex and the economic cost is too high. Therefore, the commercialization of these technologies is limited. Machine vision based intelligent vehicle distance intelligent inspection and control technology has been gaining attention to the manufacturers. On the hardware,

2.3 Intelligent Detection and Control Technology of Safety Vehicle Distance

33

the pinhole cameras 1 and 2 are used to collect the image in front of the vehicle in real time, and this image is collected by the vehicle image information processing system to realize the whole operation process of the Fig. 2.1.

2.3.1 Method for Rapid Identification of Vehicle in Front Template matching is one of the efficient and fast algorithms for vehicle recognition in the front.

2.3.1.1

Principle of Template Matching Method

Template matching is one of the main methods of image recognition. The purpose of template matching is to find the most suitable relation between the scene image to be matched and preset template image. 1. Spatial Domain Description of Template Matching Suppose the scene image is s(x, y), the image size is W × H , the template image is t(x, y), and the image size is w × h. Normally, the pixel size of the scene image is much larger than that of the template image, that are W ≥ w and H ≥ h. The basic idea of spatial domain template matching with image enhancement method of spatial domain filtering used is similar, but the numerical filter operator of image enhancement is fixed, generally smaller, and the template matching operator is used in the template image itself, generally larger size. The spatial domain filtering method basically adopts convolution or correlation method, and template matching can perform complex operations on the basis of field image and template image, and the form of operator is more flexible. To match templates in the spatial domain, the usual approach is to use a template as a filter operator to glide through the entire image, seeking the image region that matches the baseline. This process is sometimes referred to as Back Project, and the results obtained can be expressed as R(s, t, x, y) 

w−1  h−1 

f {s(x + m, y + n), t(m, n)}

(2.7)

m0 n0

Among them, x ∈ [0, W − w] and y ∈ [0, H − h]. As a result, the image size is (W − w + 1) × (H − h + 1). You can see that R(s, t, x, y) is related to the template t(x, y), where the scene image s(x, y) is related to the current location (x, y). The matching process, if the scene and the template image is fixed, the results can be abbreviated as R(x, y). Finally, by analyzing the numerical value of R(x, y), the matching position (x, y) is obtained. According to the different forms of operators

34

2 Vehicle Driving Safety Technology Based on IVP

f {·}, there are different forms R(x, y), so different template matching algorithms can be extended. (1) The first kind of similarity is the error The starting point of the first kind of algorithm is simple. It mainly calculates the error between the template image and the scene image, and its definition imitates the definition of all kinds of errors. Absolute Error Definition of absolute error SAD(x, y) is SAD(x, y) 

w−1  h−1 

|s(x + m, y + n) − t(m, n)|

(2.8)

m0 n0

Using the template area w × h, except for SAD(x, y), we can obtain a conceptually equivalent algorithm that is MSD/MSE (Mean of Absolute Difference/Error). Variance Definition of variance SSD(x, y) is SSD(x, y) 

w−1  h−1   2 s(x + m, y + n) − t(m, n)

(2.9)

m0 n0

Using the template area w × h, except for SSD(x, y), we can obtain a conceptually equivalent algorithm that is MSD/MSE (Mean of Square Difference/Error). From SSD(x, y), the normalized variance is defined by the Normalized Sum of Square Difference (NSSD) is given by 2 w−1 h−1  m0 n0 s(x + m, y + n) − t(m, n) NSSD(x, y)   w−1 h−1 2 w−1 h−1 2 m0 n0 s(x + m, y + n) · m0 n0 t(m, n)

(2.10)

It can be found that for the first kind of similarity, closer will be template and the scene image at the (x, y), the smaller the error value R(x, y). After that final image of (W − w + 1) × (H − h + 1) is obtained, the position of the template matching can be taken as long as the minimum point of the whole image is found. Since the minimum value is obtained, sequential acceleration can be performed in the process of accumulation. The first kind of algorithm is like point contrast method and has the advantages of simple calculation and fast speed. It is especially suitable for the scene illumination and the noise condition is not changed greatly. For example, the template motion matching of the previous frame and the last frame in the video, or the tracking matching of the video surveillance system. Such algorithms usually start from the

2.3 Intelligent Detection and Control Technology of Safety Vehicle Distance

35

previous matching position, and part of the previous scene image is used as a template for matching and tracking systems. At present, a series of mature MSD (Mean of Square Difference) and SSDA (Sequence Sum Difference Algorithm) algorithms have been widely used in motion vector analysis of video acceleration. (2) The second kind of similarity is cross-correlation The first disadvantage is this algorithm of illumination change and is very sensitive towards noise, especially by capturing template matching to the scene situation afterwards changed, due to illumination and noise conditions are changed, or is the template on site object occurred (not parallel moving) affine transformation. Such as zoom and rotation, then, the effect is not ideal. In order to make the template adapt to the brightness, the second kinds of algorithms are added to the additive and multiplicative properties of the object. Cross-correlation Cross correlation for matching is inspired by the SSD algorithm. Expand SSD(x, y) that is SSD(x, y) 

w−1  w−1  h−1 h−1   2  s(x + m, y + n) + [t(m, n)]2 m0 n0

−2

w−1  h−1  

m0 n0

s(x + m, y + n) · t(m, n)



(2.11)

m0 n0

As you can see, the first item after the expansion is only related to the scene image. The second item is only concerned with the template image, and it has nothing to do with the position. The third most important relation between the template and the scene image is third item. The third term is actually an expression similar to spatial domain filtering, which is a signal cross-correlation. Larger the third items, smaller will be the error, greater the similarity than smaller will be the third items, and greater the error, smaller will be the similarity. Cross correlation similarity is defined as CCOR(x, y) 

w−1  h−1    s(x + m, y + n) · t(m, n)

(2.12)

m0 n0

The relationship between normalized cross correlation and cosine coefficients Similarly, the normalized cross correlation can be obtained from the NSSD  w−1 h−1  m0 n0 s(x + m, y + n) · t(m, n) (2.13) NCCOR(x, y)   w−1 h−1 2 w−1 h−1 2 s(x + m, y + n) · t(m, n) m0 n0 m0 n0

36

2 Vehicle Driving Safety Technology Based on IVP

The image s(x, y) and image t(x, y) are considered as vector A of Euclidean linear space with w × h, defined as the inner product s·t 

w−1  h−1 

[s(m, n) · t(m, n)]

(2.14)

m0 n0

Then the Euclidean norm of the image is

s 

w−1  h−1 

s(m, n)2

(2.15)

t(m, n)2

(2.16)

m0 n0

The Euclidean norm of a template is

t 

w−1  h−1  m0 n0

Normalized cross correlation is NCCOR 

s·t  cos θ

s · t

(2.17)

where θ is the angle between s(x, y) and t(x, y) in the Euclidean linear space defining the above inner product. When s(x, y) and t(x, y) are linearly related, θ  0◦ , the two parameters are most relevant, then NCCOR  1. When s(x, y) and t(x, y) are linearly orthogonal, the two parameters are irrelevant, then NCCOR  0. From the formula (2.17), it can be seen that the scene and template matching result function R(s, t) are satisfied by 0 ≤ R(s, t) ≤ 1. From the formula (2.17), if compared to a portion of the image and the template, the emergence of a multiplicative brightness transforms of brightness, which is s(x, y)  k · t(x, y), at NCCOR  1, indicating that NCCOR can resist the multiplicative transform of brightness. Normalized cross correlation is also known as the cosine coefficient. Correlation coefficient Cross correlation can resist the multiplicative transformation of brightness, but it cannot resist the additive transformation of brightness, that is s(x, y)  k · t(x, y) + b Considering the additive variation of image brightness, cross correlation is called correlation coefficient, order

2.3 Intelligent Detection and Control Technology of Safety Vehicle Distance

1  s(x + a, y + b) w · h a0

37

w−1 h−1

s (x + m, y + n)  s(x + m, y + n) −

(2.18)

b0

1  t(a, b) w · h a0 w−1 h−1

t (m, n)  t(m, n) −

(2.19)

b0

Formulas (2.18) and (2.19) are subtracted field images and templates by mean value in the region. The correlation coefficient is defined as CCOEFF(x, y) 

w−1  h−1    s (x + m, y + n) · t (m, n)

(2.20)

m0 n0

The normalized correlation coefficient is defined as  w−1 h−1  m0 n0 s (x + m, y + n) · t (m, n) NCCOEFF(x, y)   w−1 h−1 2 w−1 h−1 2 m0 n0 s (x + m, y + n) · m0 n0 t (m, n) (2.21) Thus it can be seen,if a part of the image is compared to the template, the multiplicative and additive transformations of the brightness occur, that is s(x, y)  k · t(x, y) + b, at this NCCOEFF  1, that NCCOEFF can resist the additive transformation of brightness in addition to the multiplicative transformation of brightness. If the image is viewed as a random variable, then the denominator of the normalized correlation coefficient is the covariance of the field image and the template image, the molecular part is their corresponding variance, the similarity of correlation coefficients can be derived from the concept of the statistical correlation coefficient. The second kind of similarity is that the template is closer to the w × h region of the scene image at the point (x, y), results in the bigger the R(x, y). Exploiting after the result of (W − w + 1) × (H − h + 1) is obtained, the position of the maximum position of the image can be taken as the template matching position. Since the maximum value is sought, there is no sequential acceleration algorithm similar to the first kind of similarity in the accumulation. Compared to the first kind of similarity, second kinds of similarity index are considered robust to some certain factors such as illumination consistency change, a small part of the image changes, image of the small part is blocked, the system error, so it is widely applied in the technical field of target tracking and image registration. The cross-correlation similarity (CCOR) method is conceptually the clearest. Despite the similarity of its representation in mathematics are the most convincing, but it is relative to the first class similarity, its computation is complex, it has a certain resistance to the brightness of the multiplicative change, but for the change of additive is not very fit. The correlation coefficient (CCOEFF) method for both the multiplicative or additive variation of the brightness can be adapted to changes,

38

2 Vehicle Driving Safety Technology Based on IVP

but the calculation is the most complex, because after subtracting the mean value of digital image output has positive or negative value, therefore affected by the noise effect will be greater, more easily to misjudge. Parallel acceleration of similarity The speed of identification and speed of vehicle vision system should must meet the requirement of real-time. In other words, all recognition operations must be completed within the given time, otherwise, there is no practical value. For parallel acceleration, that is, we can calculate several kinds of similarity in the same cycle at the same time, in order to simplify the calculation. So that all the similarity which is needed to be calculated can be completed in a short time. The algorithm for parallel acceleration of similarity is formulated as ⎧  ⎪ sumS(x, y)  s(x + m, y + n) ⎪ ⎪ ⎪ m n ⎪ ⎪ ⎪ 2  ⎪ ⎪ ⎪ sumSS(x, y)  s(x + m, y + n) ⎪ ⎪ m n ⎪ ⎪ ⎨  sumT  t(m, n) (2.22) m n ⎪ ⎪ ⎪   ⎪ ⎪ sumTT  [t(m, n)]2 ⎪ ⎪ ⎪ m n ⎪ ⎪ ⎪   ⎪ ⎪ s(x + m, y + n) · t(m, n) ⎪ ⎩ sumST (x, y)  m

n

Thus obtained SSD(x, y)  sumSS(x, y) + sumTT − 2sumST (x, y) (2.23) sumSS(x, y) + sumTT − 2sumST (x, y) NSSD(x, y)  (2.24) √ sumSS(x, y) · sumTT CCOR(x, y)  sumST (x, y) (2.25) sumST (x, y) NCCOR(x, y)  √ (2.26) √ sumSS(x, y) sumTT 1 CCOEFF(x, y)  sumST (x, y) − sumS(x, y) · sumT (2.27) w·h 1 sumST (x, y) − w·h sumS(x, y) · sumT NCCOEFF(x, y)    2  1 1 sumSS(x, y) − w·h sumTT − w·h sumS(x, y) (sumT )2 (2.28) Thus, we can calculate the similarity between the first and second classes by only 5 cumulative sum.

2.3 Intelligent Detection and Control Technology of Safety Vehicle Distance

39

2. Frequency domain description of template matching In addition to the description form of spatial domain, template matching is also described in frequency domain. The above both kinds of similarity are essentially kinds of spatial domain template matching. They are widely used in target tracking, image registration and other technical fields. However, due to the extreme peak cross-correlation similarity are relatively flat, and around the level of the difference is not large, if you do not consider the second high and only consider the extremely high point, it would be easy to let the noise influences the best matching point judgment, which is not good for the image recognition of low SNR. Although there is a fast algorithm of the cross-correlation image from the spatial domain to the frequency domain transfer for benefiting to take advantage of the fast Fourier transform (FFT) algorithm, thus more quickly, but the classical template matching algorithm does not break the space domain correlation nature. The phase spectrum method is to abandon the spatial domain template matching ideas, that template and scene image are transformed into frequency domain, starting from the study of affine transform spectrum analysis, combined with the matched filter and phase filter theory of random signal processing, direct search template and affine relation field images in frequency domain. Frequency domain interpretation of normalized cross correlation Considering formula (2.13), where the A reaches its maximum position as the best result. If viewed from the point of view of the matched filter in the stochastic signal processing, the Eq. (2.13) is actually an output of the matched filter. The scene image C and the template image D are regarded as the generalized stationary process of the two-dimensional joint ergodic, and the E is the input signal which satisfies   s(x, y)  t x , y + n(x, y)

(2.29)

Among them, n(x,  y)is the noise signal, and it and t(x, y) are statistically indepenrelation. The affine transformation dent. Coordinate x , y and (x, y) satisfy affine   of image f (x, y) is called image g(x, y)  f x , y , and the condition is satisfied by          b1 a11 a12 x x x +   A + b, |A|  0 (2.30) y a21 a22 y y b2   T a11 a12 , b  b1 b2 . Among them, A  a21 a22 Under the two-dimensional condition of the image space domain, Twodimensional Fourier transformation of the image f (x, y) is defined as 

+∞ +∞ F(u, v)  F{f (x, y)}  −∞ −∞

f (x, y) · e−j(ux+vy) dxdy

(2.31)

40

2 Vehicle Driving Safety Technology Based on IVP

Its inverse transformation of Fourier is defined as −1

f (x, y)  F {F(u, v)} 

1 4π 2

+∞ +∞ F(u, v) · ej(ux+vy) dud v

(2.32)

−∞ −∞

 G(u,  v)is the transformed image spectrum, and G(u, v)  F{g(x, y)}  F f x , y . And the relation between the spectrum F(u, v) and the spectrum G(u, v) after affine transformation is obtained by the Bracewell method, it is shown as follows   1 j |A|1 [(a22 u − a21 v)b1 + (−a12 u + a11 v)b2 ] a22 u − a21 v −a12 u + a11 v G (u, v)  F e , |A| |A| |A| (2.33) Among them, j2  −1 For high-dimensional Euclidean space n , the generalized time domain vector  p  t1 t2 · · · tn is defined, which has the original function f (p), and the corresponding Fourier transform is  F(q)  f (p) · e−jp·q dp (2.34) 

Its inverse transformation of Fourier is defined as  1 f (p)  F(q) · ejp·q dq (2π )n

(2.35)



  Among them, q  ω1 ω2 · · · ωn , it is called generalized spectrum vector. p · q  t1 ω1 + t2 ω2 + · · · + tn ωn is the inner product of p and q. Affine transformation of vector p is used to obtain p  Ap + b, |A|  0

(2.36)

  For g(p)  f p  f (Ap + b), an affine transformation is completed, and the vector spectrum analysis of the affine transformation is obtained G(q)  F{f (Ap + b)} 

1 j(A−1 b)·q  −T  F A q e |A|

(2.37)

 T The upper form is not only limited to two-dimensional images p  x y and  T q  u v , but it is also suitable for high dimensional situations. Similarly, the Multidimensional Discrete Fourier transformation and inverse transformation are commonly used in digital image processing

2.3 Intelligent Detection and Control Technology of Safety Vehicle Distance

F(q) 

N−1 

p

e−2πjq· N f (p)

41

(2.38)

p0 N−1 

1

f (p)  Nn

l1

Nl

q

e2πjp· N F(q)

(2.39)

q0

  Among them, such that Discrete time vector p  m1 m2 · · · mn .   Discrete frequency vector q  k1 k2 · · · kn .   Discrete scale vector N  N1 N2 · · · Nn . N−1  p0

 

N 1 −1 N 2 −1 

...

m1 0 m2 0

p m  N11 N  q  Nk11 N

N n −1 mn 0

m2 N1

···

mn Nn

k2 N1

···

kn Nn

T

T

A similar conclusion can be obtained from formula (2.37) G(q)  F{f (Ap + b)} 

1 j2π (A−1 b)· q  −T  NF A q e |A|

(2.40)

The Fourier property of many affine transformations can be extended from the formula (2.40), among which the most noticeable is the influence of parallel shift transformation. The spectrum changes caused by parallel shift transformation in two-dimensional space are limited to phase, but have no influence on amplitude. This means that the amplitude of the spectrum before and after the parallel motion of the image is the same, which the spectral amplitude is a parallel shift invariant. It is described by F{f (x + b1 , y + b2 )}  ej(ub1 +vb2 ) F(u, v)

(2.41)

The S(u, v),T (u, v), and N (u, v) respectively are defined as spectral functions of s(x, y), t(x, y), and n(x, y). Suppose that n(x, y) is a zero expected Gauss white noise, that is N (u, v)  nw . If t(x, y) is to be recovered from s(x, y), a matched filter with maximum signal-to-noise ratio (SNR) can be used H (u, v)  T ∗ (u, v) In the formula, T ∗ (u, v) represents the conjugate function of T (u, v). After the S(u, v) passes the filter, its spectrum is

(2.42)

42

2 Vehicle Driving Safety Technology Based on IVP

R(u, v)  S(u, v)T ∗ (u, v)

(2.43)

As can be seen from the formula (2.43), the matched filter output is in fact the mutual power spectrum of the field image and the template image. According to the Wiener-Khinchine theorem, the time averaged autocorrelation function and power spectral density of the power signal are exactly a pair of Fourier transform pairs, and the mutual power spectral density of the two signals are the Fourier transform of the cross-correlation function r(x, y)   R(u, v)  F r(x, y) In the formula, r(x, y) 

 m

(2.44)

s(x + m, y + n) · t(m, n), F[·] is the Fourier trans-

n

form. Therefore, the output of the matched filter can be represented as cross correlation function in the spatial domain, which is the inverse Fourier transform of the cross power spectral density    s(x + m, y + n) · t(m, n) (2.45) r(x, y)  F−1 S(u, v)T ∗ (u, v)  m

n

In the formula, F−1 {·} are the inverse Fourier transform. Formula (2.45) is actually the molecular part of the formula (2.13), and expresses the nature of a cross correlation matched filter. Compared with Wiener filter, the disadvantage of the matched filter for identifying signals is the pursuit of maximum signal-to-noise ratio. This makes the output of the matched filter related to the energy of the input/ output signal. Therefore, “cross correlation” is seldom used to match templates. In order to match the template, it is necessary to normalize the cross-correlation using the energy of the input signal. The denominator of formula (2.13) is essentially the energy product of the field image and the template image, and the template, for example, has energy Et 

 m

|t(m, n)|2 

n

1  |T (u, v)|2 M ·N u v

(2.46)

In the formula,M  w − 1, N  h − 1 Cross power phase spectrum similarity Inspired by the idea of normalized cross correlation (NCCOR) matched filter, parallel movement transformation between image and template occurs, s(x + x0 , y + y0 )  t(x, y), according to F{f (x + b1 , y + b2 )}  ej(ub1 +vb2 ) F(u, v) There is

(2.47)

2.3 Intelligent Detection and Control Technology of Safety Vehicle Distance

T (u, v)  ej(ux0 +vy0 ) S(u, v)

43

(2.48)

Substitute it in the formula (2.43), so there is R(u, v)  e−j(ux0 +vy0 ) |S(u, v)|2

(2.49)

From the above formula we can find the cross power spectrum before and after the parallel shift transform, and its amplitude spectrum is only related to the amplitude of the field image, the phase spectrum is only related to the parallel moving vector T  b  m0 n0 . m0 and n0 are phase parallel movement quantities corresponding to parallel moving quantities b1 and b2 in time domain. Inspired by this, a nonlinear phase filter is added to each other based on the mutual power spectrum matching filter to extract the phase spectrum, and the mutual power phase spectrum is defined by P(u, v) 

S(u, v) T ∗ (u, v) R(u, v)  ·  ejϕr (u,v)  ej[ϕs (u,v)−ϕt (u,v)] |R(u, v)| |S(u, v)| |T (u, v)|

(2.50)

Among them, ϕs (u, v), ϕt (u, v) and ϕr (u, v) are respectively phase spectra of s(x, y), t(x, y) and r(x, y). In order to highlight the magnitude of the phase difference, the mutual power phase spectrum is transformed into the spatial domain, PSP(x, y) is defined as the mutual power phase spectrum similarity, which is the inverse Fourier transform of the mutual power phase spectrum P(u, v), that is PSP(x, y)  F−1 {P(u, v)}

(2.51)

Obviously, when s(x + x0 , y + y0 )  t(x, y), P(u, v)  e−j(ux0 +vy0 ) , this is a complex function with amplitude 1, and the mutual power phase spectrum similarity is   PSP(x, y)  F−1 e−j(ux0 +vy0 )  δ(x − x0 , y − y0 ) 

1 x  x0 and y  y0 0 x  x0 or y  y0

(2.52)

In above mentioned equation, δ is the Dirac unit pulse function. The mutual power  T phase spectrum similarity reaches the maximum at x0 y0 . Through the comparison of formula (2.13) and cross-correlation similarity formula (2.52) cross power phase spectrum similarity indicates that, in the ideal case will reach the maximum of 1 at (x0 , y0 ), the difference between the two is that the cross-correlation similarity in the extreme around fell flat, and cross power spectral density in the extreme conditions around the sharp decline of similarity (Dirac is a unit pulse function).

44

2 Vehicle Driving Safety Technology Based on IVP

Fourier-Mellin similarity The similarity operator mainly uses parallel moving template in scene image to do similar calculation. From the point of view of geometric transformation, if the template and the scene are regarded as the images before and after geometric transformation, the similarity operator can only extract the part of the parallel moving transform, and can only deal with the s(x + x0 , y + y0 )  k · t(x, y). When the scene image is scaled and rotated with respect to the template, the equation can be expressed as s(ax cos θ − ay sin θ + x0 , ax sin θ + ay cos θ + y0 )  k · t(x, y)

(2.53)

All the similarity operators will fail, and at this time we often require operators like HU moments to resist rotation, scaling, and parallel motion. Starting from the similarity of the mutual power phase spectrum, we can further explore a more general affine transform matched filter in the frequency domain. The mutual power phase spectrum similarity can overcome the influence of large scale scaling and rotation on template matching, and improve the matching filter effect and the accuracy and success rate of template matching. Among them, the log polar transformation is one of the methods used to solve this problem. The log polar transformation is also a geometric transformation, but it is nonlinear, the formula for the transformation of coordinates would be ⎧   ⎨ ρ(x, y)  ln r  1 ln x2 + y2 2 (2.54) ⎩ ϕ(x, y)  arctan y x where, −π < θ (x, y) ≤ π

 T Assuming that the coordinate p  x y is rotated, it will get      x x cos θ − sin θ p   a · Rp  a y y sin θ cos θ

Before and after coordinate transformation, the relation is written as ⎧  " 1    1 ! 2  2 ⎪ ⎨ ρ x , y  ln x + y  ln a2 x2 + y2  ln a + ρ(x, y) 2 2 "  !   y ⎪ ⎩ ϕ x , y  arctan  arctan tan θ + arctan y  θ + ϕ(x, y) x x

(2.55)

Which is translated to ⎡  ⎤     ρ p ρ(p) ⎣   ⎦  ln a + θ ϕ(p) ϕ p

(2.56)

2.3 Intelligent Detection and Control Technology of Safety Vehicle Distance

45

A conclusion is drawn that the rotation and scaling of the original space domain can be transformed into a parallel motion in polar logarithmic coordinates. In this  T way, we can accurately detect the best parallel motion ρ0 ϕ0 by using various similarity including cross power spectrum density similarity, and the best matching a  eρ0 scaling and rotation amount are . θ  ϕ0 The optimum matching parallel motion can be used to estimate the best matching scaling and rotation between the template and the field image in the polar logarithmic space. The concrete steps are as follows (1) To obtain the power spectral density similarity between the scene image and the template image. (2) Phase matching is used to estimate the best matching parallel moving vector T  m0 n0 , and the cross power phase spectrum similarity is also considered. (3) The amplitude spectrum is transformed by log polar coordinate, and s(ρ, ϕ) and t(ρ, ϕ) are obtained. (4) Calculate the normalized power spectral density of s(ρ, ϕ) and t(ρ, ϕ). T  (5) Estimate the best matching parallel moving vector ρ0 ϕ0 . (6) Find the best matching scaling a and the rotation amount θ , so as to complete the matching. If the cross spectral density similarity of the image f (ρ, ϕ) after the transformation of the log polar coordinates is calculated, the two-dimensional Fourier transformation of the f (ρ, ϕ) is performed to get +∞ +π F(μ, ν)  F{f (ρ, ϕ)} 

f (ρ, ϕ) · e−j(μρ+νϕ) d ϕd ρ

(2.57)

−∞ −π

Consider ρ  ln r, which can be obtained after substitution +∞ +π F(μ, ν) 

f (ln r, ϕ) · e−j(μ ln r+νϕ) d ϕd (ln r)

−∞ −π +∞



e

−jμ ln r

+π

f (ln r, ϕ)e−jνϕ d ϕ

−π

0

+∞ +π dr −jμ r f (ln r, ϕ)e−jνϕ d ϕ  r 0

dr r (2.58)

−π

# +π where, fˆ (ν)  −π f (ln r, ϕ)e−jνϕ d ϕ is the function of one-dimensional Fourier transform in the angle, called the ring Fourier transform.

46

2 Vehicle Driving Safety Technology Based on IVP

# +∞ F(μ, ν)  0 r −jμ fˆ (ν) drr is the Mellin transform when s  −jμ, that is # +∞ F(s)  0 r s f (t) drr . The radial Mellin transform is called, so this transform can be collectively referred to as the toroidal Fourier- radial Mellin transform. In the usual sense, the Fourier-Mellin transform is defined +∞ +π dr r s f (r, ϕ)e−jνϕ d ϕ F(s, ν)  r 0

(2.59)

−π

The transformation is actually used to unify the expression of various rotation/ scaling/ parallel motion invariant moments. When s is an integer greater than 1 and ν  0, F(s, ν) is a “zoom/rotate/parallel mobile invariant moments”; or, when s arbitrary complex value, at the same time ν takes any real value, F(s, ν) is the generalized “rotation/zoom/parallel mobile invariant moments”. Thus, combining the log polar transform and Fourier transform, is a purely imaginary Fourier-Mellin transform. If we ignore the presence of ln r in the expression f (ln r, ϕ), it will inevitably lead to confusion between the two concepts.

2.3.1.2

Identifying Ahead Vehicles

1. Build the Vehicle Background Template Database Collecting and organizing templates is the basic job of template matching. By using the rapidity of template matching and the rotation/zoom/parallel motion invariant moments, a vehicle background template database is built up in vehicle control system. As shown in Fig. 2.12, a set of templates built up for the collection of background images of various types of vehicles is listed in turn.

Fig. 2.12 Examples of vehicle background images database

2.3 Intelligent Detection and Control Technology of Safety Vehicle Distance

47

2. Real-time collection of scene images in front of vehicles The lane can be basically divided into three types, the right lane, the middle lane and the left lane, of which there may be only one lane, or there may be more than one lane. By using the template matching algorithm, it is unnecessary to consider whether the vehicle is in the specific lane at any time, and there is no need to consider the lane indication sign on the lane. Taking the binocular vision (consisting of 1 and 2 of pinhole cameras, see Fig. 2.1), the scene image of the vehicle is collected in real time, which is called the main field of view image. In Fig. 2.13, it’s a real-time image after the synthesis from the binocular sense of left and right view in front of the vehicle. The main field image is intercepted according to the size (specification) which is most conducive to analysis and operation. As shown in Fig. 2.14, a few main field images are enumerated, and the dashed line frames in the picture are the binocular main field observation windows. The selection of the window just makes the operation focus on the vehicle identification within 30 to 200 m distance from the vehicle. 3. Fast identification algorithm for forward vehicle If the size of the template is w × h, w and h are the columns and rows of the template respectively. The size of the main field view window is W × H , W and H are the columns and rows of the window. Each time a template is selected from the template database to match with the observation window, and the template matching algorithm is used to rapidly determine whether there is a vehicle in front. Since the main view field observation window has been greatly reduced from the entire view, the search for the target does not have to start from one to the top left of the view pixel (0, 0), so the computation speed can be increased by an order of magnitude.

Fig. 2.13 Main field observation windows

48

2 Vehicle Driving Safety Technology Based on IVP

Fig. 2.14 Main field observation windows

Each matching process, uses the coarse matching and fine matching method combined with the search, that is to say, the coarse matching stage uses a larger step jump calculation of some similarity. Let the fine matching routine after the best matching position, if the matched position is near the coarse matching, than it can improve the speed of matching. At the same time, random walk method is also introduced, also named as a variable step method, in this method the distance jump away to search the starting point, at the same time the position of the search may also be up and down around the turns, began to walk from the starting point perpendicular and horizontal staggered. For example, a typical complex walk is most famous for the analysis of the motion vector MPEG a “diamond search algorithm” or “diamond search algorithm” (Diamond Search). The matching similarity described here can be applied to any of the above similarity. The experimental results show that the fast template matching algorithm is used to identify the time on the front of the vehicle is less than 30 ms for long distance vehicle recognition time, and for short distance vehicle recognition time is much shorter, such as the distance of 200 m vehicle recognition time is 5 ms. See the black dashed frame area shown in Fig. 2.15.

2.3 Intelligent Detection and Control Technology of Safety Vehicle Distance

49

Fig. 2.15 Long distance vehicle identification process

2.3.2 Vehicle Distance Measurement Once the vehicle is identified in front, it is a very important parameter to control the distance between the vehicle itself and the vehicle ahead.

2.3.2.1

Parallax Principle

Binocular vision is a machine structure that simulates binocular vision of animals. Double vision geometry is the theoretical foundations of binocular vision sensing. To facilitate the discussion, it is assumed that the two cameras are parallel to each other on a horizontal plane, i.e., the optical axes of the two cameras are parallel to each other. As shown in Fig. 2.6, the virtual imaging planes of the two cameras overlap with each other to become a common virtual imaging plane . Ol and Or are respectively the optical center of left and right cameras (i.e., the center of the lens, referred to as the photocenter), the logical focus of the two cameras is f . It is assumed that the point P on the object is projected at Pl and Pr on the image plane of the left and right camera (virtual imaging plane ), respectively. Make a vertical line from P to line Ol Or . AP and OP are the intersection of the perpendicular line with the virtual imaging plane  and even the line Ol Or . Over Ol and Or make two vertical lines to the virtual imaging plane , which intersect the imaginary imaging plane  at point Al and Ar respectively. The formulas can be derived from the similar triangles in the Fig. 2.16 |Pr AP | |PAP |  |POP | |Or OP |

(2.60)

50

2 Vehicle Driving Safety Technology Based on IVP

Fig. 2.16 Parallax ranging principle diagram

|Pl AP | |PAP |  |POP | |Ol OP |

(2.61)

Here, |·| represents the length of the line between two points. If the |POP |  a, |AP OP |  f˜ , |Ol Or |  b, |Al Pl |  l, |Ar Pr |  r, and |Pr AP |  c, than formulas (2.60) and (2.61) can be expressed as a − f˜ c  a r+c a − f˜ b−l+r+c  a b+r+c

(2.62) (2.63)

If the formulas (2.62) and (2.63) arranged, then we can obtain c

b·r −r l−r

(2.64)

a

b · f˜ l−r

(2.65)

Put it in (2.62), we obtain

In the formula, l − r is called binocular parallax. The depth information of the scene is the distance a to be measured. It has a relationship with the two camera’s optical center distance b, focal length f˜ and two camera imaging parallax l − r. Since the center distance b of the two cameras and the focal length f˜ of the imaging are calibrated by the camera, they are known in advance. Therefore, the parallax l − r of the two cameras is the only factor that can determine the depth of the scene. The focal length f˜ of the image and the parallax l −r of the two cameras are calculated as pixels. The optical center distance b between the two cameras is calculated as the actual distance (m).

2.3 Intelligent Detection and Control Technology of Safety Vehicle Distance

51

In fact, when binocular vision sensing is used, there is a complex functional relation between the measurement error of the measured point and the angle of the two camera axis. When the angle between the two cameras optic axis is fixed, the distance between the coordinates of the measured points in the world coordinate system and the camera is larger, and the detection error of the detection point distance is greater. At present, when the binocular pinhole camera is set up in the vehicle vision system, the parallax principle has been considered for distance measurement, so during the installation of two pinhole camera it is required that the two-pinhole camera optical axis can be parallel to each other and perpendicular to the optical center of the connection line. Because of the geometric characteristics of the two pinhole cameras in a binocular vision system, binocular parallax principle can be used directly in the detection of vehicle distance.

2.3.2.2

Vehicle Distance Calculation

Using the epipolar constraint principle, a common feature point of the matched target is quickly determined. According to the corresponding feature points Pl and Pr in the virtual imaging plane, the distance between the common feature points and the vehicle is calculated directly by using the parallax principle. In vehicle binocular vision system, distance between two-pinhole camera optical center and the imaging logic focal length (determined by camera parameter calibration) is a known value. The conversion formula of virtual imaging plane coordinate system to computer image coordinate system if

y x + cx , j  f + cy , sx  dy dx dx dy

(2.66)

The image coordinates are converted to the virtual imaging plane coordinate system, that is, the virtual imaging plane coordinates of the points Al , Pl , Ar and Pr can be obtained by the conversion formula, that is ⎧ ⎪ Al  (xAl , yAl ) ⎪ ⎪ ⎪ ⎨ P  (x , y ) l Pl Pl (2.67) ⎪  A (x r Ar , yAr ) ⎪ ⎪ ⎪ ⎩ P  (x , y ) r Pr Pr Considering yAl  yPl  yAr  yPr , the binocular parallax can thus be obtained, that is l − r  |Al Pl | − |Ar Pr |  |xAl − xPl | − |xAr − xPr |

(2.68)

52

2 Vehicle Driving Safety Technology Based on IVP

In formula (2.66), f is the effective focal length of the camera optics, (dx, dy) are the distance between the pixels in of x and y directions in the image plane, and sx is the aspect ratio of the image. cx and cy is the pixel coordinates of optical  center o in computer imaging plane, that is the optical center coordinates cx , cy . The formula (2.68) with known binocular camera optical center distance b and focal distance f˜ is substituted into the formula (2.65), the depth information of the scenery can be obtained directly which is the value of the vehicle from distance a. It must be noted that during calculation process the binocular camera optical center with distance b is calculated as m, and the remaining two quantities in formula (2.65), i.e., the parallax l − r and imaging focal length f˜ are measured in pixel units.

2.3.3 Driving Decision Based on identification results and the determination of the presence of the vehicle in front of the vehicle is considered, it is also necessary to know about the lane if the vehicle is in front of the vehicle than it is considered whether the vehicle is in the same lane as the vehicle itself. If you are in the same lane, control your speed accurately. If the vehicle is in front of their left and right lanes, then it should pay attention so that vehicles cannot make any collision or accident with other lane vehicles. Due to this reason, the lane identification problem would be solved immediately.

2.3.3.1

Vehicle Lane Identification Algorithm

The vehicle lane identification algorithm is as follows. 1. Seek the row projection center of the ahead vehicle (1) The images collected in real time are rotated, in order to overcome the abnormal image caused by the uneven road. As shown in Fig. 2.17, the Fig. 2.17a is the original image, and Fig. 2.17b is an image corrected by the rotation transformation. (2) When the vehicle visual sense system uses the template w×h to find the matching area in the main view window, the row projection center of the vehicle in front is calculated by the template matching region. As shown in Fig. 2.18, a horizontal line is made for the matching area which intersects the two intersection points (il , jl ) and (ir , jr ) on the left and right edges. To do a vertical bisector of two intersecting lines, the intersection point between the vertical bisector and the coordinate axis i is the row projection center M of the vehicle in front, and is given by   il − ir ,0 M (i, j)  il − 2

(2.69)

2.3 Intelligent Detection and Control Technology of Safety Vehicle Distance

53

Fig. 2.17 Jump property of grey level of road sign line

( il , jl )

Fig. 2.18 Geometric significance of row projection center in front of vehicle

( ir , jr ) j

i

M

0

2. Detection of mark line in carriageway (1) After the rectified image shown in Fig. 2.17b passes through the two-valued processing, as shown in Fig. 2.19, the mark line of carriageway are highlighted. (2) Continue to deal with skeletonization of two-valued images. The so-called skeletonization is a method of mathematical morphology, that is, in the process of etching, grasp a certain width, so that the “corrosion” results retain the skeleton of the identified feature blocks. After two-valued images are treated with skeletonization, the resultant image is shown in Fig. 2.20. [Methods of mathematical morphology] The basis of mathematical morphology and its language belong to the class of set theory. Mathematical morphology can simplify image data, maintain their basic shape features, and remove irrelevant structures. Mathematical morphology is made up of a set of morphological algebraic operators. It has four basic operations, including

54

2 Vehicle Driving Safety Technology Based on IVP

Fig. 2.19 Mark line image of two-valued carriageway

Fig. 2.20 Image after skeletonization

expansion, corrosion, opening and closing. Mathematical morphology can be divided into two categories: two-valued morphology and grayscale morphology. They have their own characteristics in two-valued images and gray images. The morphology applied to two-valued images is called two-valued morphology, and the morphology used in grayscale images is called grayscale morphology. Based on these basic operations, they can be deduced and combined into various mathematical morphological practical algorithms. They can be used to analyze and process the image shape and structure, including image segmentation, feature extraction, edge detection, image filtering, image enhancement and restoration, etc. The algorithm of mathematical morphology has the structure of natural parallel implementation, and realizes the parallel of morphological analysis and processing algorithm, which greatly improves the speed of image analysis and processing. The so-called structural elements are the probes used in mathematical morphology to collect information about images. When the probe moves continuously in the image, the relation between each part of the image is examined, and the structural features of the image are understood. As the structural elements of the probe, it can

2.3 Intelligent Detection and Control Technology of Safety Vehicle Distance

55

directly carry knowledge, such as shape, size, and even add gray and chrominance information to detect and study the structural features of the image. Dilation is the process of merging all the points of contact with an object into the object. The result is that the area of the object is increased by a corresponding number of points. If the object is round, its diameter will increase by two pixels each time when it expands. If two objects are separated by less than three pixels at any point in a certain point, they will be connected at that point. The expansion object in the two-valued morphology is the set, and expands the two-valued image with twovalued structure elements. The result is that the elements of the structure are moved, so that the intersection of the two points is not empty, forming a new set. Grayscale morphology is a natural extension of two valued mathematical morphology to gray scale images. The object of grayscale morphology is not set, but the image function. In grayscale morphology, intersection and merge operations are used in the twovalued morphology and they are replaced by the maximum and minimum extremum operations respectively. The expansion process of the gray image can be calculated directly from the gray level image function, and structure element calculation of dilation is performed by one point with one point. When calculating, it involves the gray value and the element value of the structure around it. In fact, this is the sum of the gray values of the corresponding points in the local range and the elements of the structure, where the maximum value is selected, so after the expansion operation, the edge is extended. The so-called corrosion operations are used to eliminate all boundary points of the object of a process, the result is to make the remaining objects along its perimeter than the original object of a small area pixels. If the object is round, it will reduce the diameter of two pixels in each pixel after corrosion, if objects connected at some point in an arbitrary direction is less than three pixels, then the object after a corrosion will be split into two objects at this point. The two-valued image is etched with two-valued structure elements, and the resultant structuring elements are shifted, so that the two-valued structure elements are confined in the two-valued images, and all the points establish a new set. The corrosion process of the gray image can also be calculated directly from the gray level function of image and structure elements, and the corrosion operations are also performed by one point with one point. The operation result of a point is the difference between the gray value of the corresponding point in a local range and the element in the structure, and the minimum value is selected. After the operation of corrosion, the gray value of the relatively large edge will be reduced, so that the edge will shrink to the region where the gray value is high. The so-called open operation is defined as the operation in which first corrosion, then expansion occurs. It has the function of removing small objects, separating objects in a slender place, and smoothing the boundaries of larger objects. In closed operation, the first expansion, then corrosion, this process is called closed operation. It has the function of filling small holes in objects, connecting adjacent objects and smoothing boundaries.

56

2 Vehicle Driving Safety Technology Based on IVP

The method based on the mathematical morphology of the multi structure elements carries out the skeletonization operation of the image, and the specific process is as follows: The first step is to select multiple structuring elements. The central point of the image is the coordinate origin, and the square structure element matrix is Bi  {f (x + x0 , y + y0 ), θi  i × α|−N ≤ x0 , y0 ≤ N }

(2.70)

Among them, Bi  Bi (s, t), i  0, 1, 2, . . . 4N − 1, N is the natural

number, (s, t) is the two-dimensional pixel matrix having coordinate, α  180◦ 4N , and θi is the direction angle formed by the structuring element in the matrix, referred to as the direction angle. The second step is to transform the image data. The gray value f  f (x, y) of the point (x, y) is input to the image from the structuring element B  B(s, t), and the operation f ⊕ B of gray expansion is carried out. The definition formula of expansion operation is $ (f ⊕ B)(x, y)  max{f (s − x, t − y) + B(s, t)$(s − x, t − y) ∈ Df and (s, t) ∈ DB } (2.71) Then the structure elements of B are used to perform grayscale corrosion operations f ΘB on the input images of f , and the definition formula of corrosion operations is given by $ (f B)(x, y)  min{f (x + s, y + t) − B(s, t)$(s + x, t + y) ∈ Df and (s, t) ∈ DB } (2.72) In the above two formulas, Df and DB are defined domains of f and B respectively. Then the expansion and corrosion operations are combined to obtain morphological gradients gi (f ), its operation formula is gi (f )  (f ⊕ B) − (f B)

(2.73)

Among them, i corresponds to the value of i in Bi . The third step is to synthesize the morphological gradient A with weighted, and to obtain the morphological gradient after synthesis g(f ˆ )

M 

ωi gi (f )

(2.74)

i1

Among them, i  1, 2, . . . M , M is the number of square structure elements, and ωi is the weight of different angles in edge detection. The fourth step is to synthesize the morphological gradient g(f ˆ ) by skeletonization based on the principle of statistics.

2.3 Intelligent Detection and Control Technology of Safety Vehicle Distance

57

First, calculate the average gray value A¯ gray of image frames, the average value of using 3 × 3 pixel area as a unit to calculate the average gray level main of nine pixels ai in this region 1 ai 9 i1 9

main 

(2.75)

Then, the mean variance eA of the pixel region gray scale is calculated one by one % & 9 & (2.76) eA  ' (main − ai )2 i1

Finally, the gray mean variance eA of the pixel region is compared with the average gray value A¯ gray of the gray image frame one by one, and the pixel region is determined to be 1 or 0 according to the following formula ⎧ ⎪ ⎪ ⎨ 1 A¯ gray ≤ n · eA (2.77) F(x, y)  ⎪ ⎪ ⎩ 0 A¯ gray > n · eA Among them, F(x, y) is the “skeletonization” image corresponding to the gray image f (x, y), and the n is multiple, determined by experiment. (3) Hough transformation of the image after skeletonization is carried out in order to obtain the traffic mark line (straight line) and determine the coordinate of the key points of the traffic sign line. The Hough transform, is an important method to identify the geometry of image processing, which has been widely used in image processing, Hough transform is not affected by the rotation of the graphics, easy to transform geometry. The simplest Hough transform is to recognize a straight line in an image. [Hough transform principle] In a rectangular coordinate system x − y, the equation of line can be expressed as y  kx + b

(2.78)

For a definite point (x0 , y0 ) in a straight line, then equation can be written as y0  kx0 + b

(2.79)

This is a straight line corresponding to the parameter plane k − b. In other words, a point in the image corresponds to a straight line on the parameter plane, and a point on the parameter plane corresponds to a straight line in the image.

58

2 Vehicle Driving Safety Technology Based on IVP

In order to detect the straight lines on the image, all the effective points on the image can be transformed into parameters, and a point on the image plane corresponds to a line on the parameter plane, the final line of the detected image must be the point at which the line in the parameter plane intersects more. In view of the fact that the most direct lines are the most difficult to work on, the methods of analysis often fail. At this point, people can solve the problem by statistical characteristics, that is, by accumulating the “parameter plane” line, the larger accumulated value is counted as the peak point. For example, if there are three straight lines on the image plane, then, when the parameters are superimposed on the parameter plane, you will see that the accumulator has three peak intersections, and so on. In practical applications, linear equations in the form of y  kx + b cannot express straight lines in the form of x  c, because the slope of the line is infinite at this point. Therefore, it is necessary to use polar coordinate parameters to express lines, that is, line x − y can also be expressed as ρ  x cos θ + y sin θ

(2.80)

The geometric meaning of the formula (2.80) is shown in Fig. 2.21. In this way, a point on x − y corresponds to a curve on the parameter plane ρ − θ , and then it can be inferred that the parameter plane ρ − θ has the intersection points of more curves correspond to a straight line on the plane x − y. The classic algorithm of Hough transform is: The first step initializes a buffer ρ − θ to record the accumulated values of the parameter plane and initialize all of its data to 0. The second step, for each valid point on the image (such as boundary points, pixels with a pixel value of 1 in the two-valued image, etc.), would find the curve corresponding to the parameter plane, add the value of the point through the curve by 1. For example, the count result of buffer ρ − θ after the Hough transformation of the image shown in Fig. 2.20 is shown in Fig. 2.22. The third step is to determine a threshold, when the maximum accumulated value exceeds a 50% of threshold, then position of relatively larger accumulated value can

Fig. 2.21 Polar coordinate parameter linear equation

y

ρ θ 0

x

2.3 Intelligent Detection and Control Technology of Safety Vehicle Distance

59

Fig. 2.22 Count result of buffer ρ − θ

be found in the parameter plane i.e. see white hollow block in the picture above, which shows that the point on a parameter plane ρ − θ has been detected in the original image corresponding to a straight line as detected. The range of inhibition is also considered to improve the Neighborhood Suppression used in the algorithm. As shown in Fig. 2.22, some of the detected parameter points are closer, which indicates that there are two straight lines of approximately the same slope and intercept in the plane x −y. These are detected separately because two-valued images are coarse, thin, or noisy, and should actually be considered the same straight line. In order to prevent the occurrence of such a situation, classic “Neighborhood Suppression” algorithm can be used to find a larger point which are accumulated around a range of values which are cleared, avoid nearby more than two points become larger. In the fourth step, the Hough transform corresponds to the linear equation without head and tail, and detecting line segments on the image is often more meaningful than just linear positioning. You can trace the pixels on the image along the detected line equation, starting at the effective point as the starting point of the line segment, until the no effective point is found at the end of the line segment. It must be pointed out that the length of the line segment detected by the statistics is discarded too short because the short line segments often correspond to noise and error detection. In addition, for the lane dashed line, the maximum separation problem should be considered. When a line segment is detected, if two segments on the same line are spaced smaller than the maximum interval, these are merged into the same line segment without separation. On the other hand, if the line segment exceeds

60

2 Vehicle Driving Safety Technology Based on IVP

the maximum distance, the two line segments are processed separately, otherwise the indication dummy segment is discarded and the error is caused. As shown in Fig. 2.23, after the Hough transformation of the skeletonization image, the line marking direction and the key position of the image on the corrected image are automatically identified. (4) To distinguish non-traffic sign line, namely according to the marking to distinguish traffic sign line or non-traffic sign line, as shown in Fig. 2.23 transverse signs from the crosswalk line to the coordinate j invariant or very small changes, only i is mobile, can easily be distinguished. 3. Confirm the carriageway of the vehicle ahead (1) Cover the lower edge of the matching area (see Fig. 2.24), make a horizontal line, and find the intersection points a and b of the traffic sign line on both sides. (2) Compare the i axis coordinate M (i) of M with the i axis coordinates of intersection points a and b.

Fig. 2.23 Skeletonization of images is carried out by Hough transform

j

Fig. 2.24 Intersection of the lower edge of the matching area and the traffic sign line a

i

b

M

0

2.3 Intelligent Detection and Control Technology of Safety Vehicle Distance

61

The i axis coordinates of a and b are a(i)  ia and b(i)  ib , respectively If il − ir < ia 2

(2.81)

b(i) < M (i) < a(i)

(2.82)

ib < il − That is

It shows that the vehicle ahead is in the same lane as the vehicle itself. If il −

il − ir > ia 2

(2.83)

That is M (i) > a(i)

(2.84)

It shows that the vehicle ahead is on the left side of the left marking line, or the left lane. If il −

il − ir < ib 2

(2.85)

That is M (i) < b(i)

(2.86)

It shows that the vehicle ahead is on the right side of the right marking line, that is, the right lane.

2.3.3.2

Vehicle Distance Control

A vehicle distance control strategy is determined based on the location of the identified vehicle. 1. The vehicles are in the same lane In front of the vehicle and its own vehicle on the same lane, we must control the speed to keep the safe distance between the two vehicles. When the vehicle mounted visual sense system has an intelligent speed linkage control device, the distance S of the safety vehicle can be calculated according to the following formula

62

2 Vehicle Driving Safety Technology Based on IVP

S

V2 2|a|

(2.87)

In the formula, A is the current speed of its own vehicle, and B is a negative acceleration, for example ( 2)     V  139(km) as the When V  120 km h−1 and a  −4 m s−2 , take S  2|a| safe distance. ( 2)     V  62(km) as the When V  80 km h−1 and a  −4 m s−2 , take S  2|a| safe distance, etc. Here, · takes integers from up. When the vehicle vision system does not have the speed intelligent linkage control device, the control of safe distance S should consider the lag factor of human reflection, that is S

V2 + Vτ 2|a|

(2.88)

In the formula, τ is the artificial reflection lag time, generally takes τ  3 s, for example ( 2)     V + V τ  239(km) When V  120 km h−1 and a  −4 m s−2 , take S  2|a| as the safe distance. ( 2)     V + V τ  129(km) When V  80 km h−1 and a  −4 m s−2 , take S  2|a| as the safe distance, etc. 2. Vehicles are divided into two lanes adjacent to each other When the vehicle in front is on the left or right lane of the own vehicle, the vehicle vision system will implement the automatic monitoring algorithm. Because, with the aid of detection technology, the distance between the detected vehicle and its own is known, so once these vehicles enter the driveway of their own vehicles due to lane change, the technology can be processed according to the safety distance control strategy algorithm in the same lane at the same time.

2.4 Intelligent Technology for Preventing Vehicle from Impacting Pedestrians During driving, vehicles often encounter pedestrians or other moving objects to cross the road. Because the driver’s carelessness, the vehicle will often lead to disaster, especially when someone is through the road in the countryside. To eliminate such traffic accidents thoroughly, it is very important to improve the automation and intelligence level of vehicles.

2.4 Intelligent Technology for Preventing Vehicle from Impacting Pedestrians

63

Intelligent techniques and algorithms for vehicle crash avoidance can be varied. One of the most critical points is to achieve accurate and rapid identification of moving objects such as pedestrians. This chapter introduces a method of fast recognition of moving objects in complex background by means of gray scale difference. This method makes full use of the moving characteristics of moving object in two consecutive frames, and can recognize the moving object accurately and quickly.

2.4.1 Algorithm Principle Like the intelligent identification technique for obstacles and pits, the scene images in front of the road are collected in real time by means of on-board pinhole cameras 1 and 2 (see Figs. 2.1 and 2.6). When the vehicle binocular vision sensing device collects the scene image in advance, the current image is compared with the image collected at the previous moment. Figure 2.25 shows a view of the front of a pedestrian passing from right to left when the vehicle binocular vision sensing device is combined with the left and right views. Figure 2.26 shows a binocular sense device of a vehicle, and then captures the view of a rider in front of the vehicle passing through the road from left to right.

Fig. 2.25 Scene of a pedestrian passing right and left through the road

Fig. 2.26 Cyclists cross the road from left to right

64

2 Vehicle Driving Safety Technology Based on IVP

The so-called “contrast” is the spatial alignment of the image sensor from the same scene background, including the two moving target images, in order to determine the relative parallel movement of the objects in the two pictures. First, each frame of two-dimensional planar image is meshed, see Fig. 2.27, which is divided into M1 × M2 square pixels, referred to as cells. Assume that each cell contains N1 × N2 pixels, and the gray level Fuv of each cell is Fuv 

u+N 1 v+N 2 1 f (i, j) N1 × N2 iu jv

(2.89)

In Eq. (2.89), f (i, j) is the gray value of the pixel (i, j), 0 ≤ f (i, j) ≤ 254, 0 ≤ Fuv ≤ 254, 0 ≤ u ≤ M1 − N1 , 0 ≤ v ≤ M2 − N2 . Because between two consecutive frames in the same context, in addition to the moving target and the surrounding areas, other areas of the gray value is relatively little change, therefore, in the digital image, select the N1 × N2 template, such as 3 × 3 or 5 × 5 pixels etc., the corresponding parts of the two frames are compared with the size of the template. According to the difference between them, the moving target area in the image can be determined. Suppose there is a continuous collection of two frames amages with a size of M1 × M2 , and the size of the template is 3 × 3, which is scanned sequentially, the (t) (t−1) of the current frame is compared with the unit Fpq of the previous frame unit Fpq image, can get $ $ $ (t) (t−1) $ (2.90) − Fpq Drs  $Fpq $; r, s  −1, 0, 1 In formula, 0 ≤ p ≤ M1 − N1 − 1,0 ≤ q ≤ M2 − N2 − 1 The Drs obtained by the formula (2.90) can be used as a basis for judging whether there is moving object in the image of the scene in front.

Fig. 2.27 Sketch map of image gridding

u Fuv

v

M1

M2

2.4 Intelligent Technology for Preventing Vehicle from Impacting Pedestrians

65

2.4.2 Threshold Extraction In order to realize the accurate decision, the decision threshold is still needed. In view of the sequence of real-time image acquisition, even the same scenery will because of fog, wind, sunlight and other natural environment in a different color contrast, i.e. after two frames may be in the scene the same part to different gray value, gray threshold for each image to calculate the frame to determine the gray level difference is acceptable at what level. Threshold here, you can use the OTSU algorithm. The OTSU algorithm divides the histogram of the image into two parts: the target and the background with the best threshold, so that the variance between the two classes is taken as the maximum, that is, the maximum separation. Set the image gray level for 1 ∼ L (such as L  255, gray level is 1 ∼ 255), gray number class l has pixel count nl (l ∈ [1, L]), the total number of pixel as L 

N

nl

(2.91)

l1

Then the probability of occurrence of grayscale pixels at level l is

Pl  nl N

(2.92)

If the gray threshold is k, the gray level of the image can be divided into two groups according to the threshold: C0 and C1 C0  {1, 2, . . . , k} C1  {k + 1, k + 2, . . . , G}

(2.93)

The total average gray level of the image is μ

L 

lPl

(2.94)

l1

The average gray level and the number of pixels of the class C0 are μ0  μ(k) 

k 

lPl

(2.95)

l1

And N0 

k  l1

nl

(2.96)

66

2 Vehicle Driving Safety Technology Based on IVP

The average gray level and the number of pixels of the class C1 are μ1  μ − μ0

(2.97)

N1  N − N0

(2.98)

And

In both part, the image proportions are w0  w(k) 

k 

Pl

(2.99)

l1

And w1  1 − w0

(2.100)

After the normalization of C0 and C1 , can get



μ¯ 0  μ0 w0 , μ¯ 1  μ1 w1

(2.101)

Then, the total image mean can be reduced to μ  w0 μ¯ 0 + w1 μ¯ 1

(2.102)

The variance between classes is σ 2 (k)  w0 (μ − μ¯ 0 )2 + w1 (μ − μ¯ 1 )2  w0 w1 (μ¯ 0 − μ¯ 1 )2

(2.103)

Can be reduced to σ 2 (k) 

[μw(k) − μ(k)]2 w(k)[1 − w(k)]

(2.104)

In the formula, the change of k from 1 ∼ L makes σ 2 (k) reach the maximum k, which is the best threshold. σ 2 (k) is called the target selection function.

2.4.3 Similarity Measurement Since the image acquisition process introduces a variety of error factors, almost no region can be found in the two images to achieve complete agreement. Therefore, when comparing the two pictures, we need to measure the similarity.

2.4 Intelligent Technology for Preventing Vehicle from Impacting Pedestrians

67

Considering any corresponding subgraph in the two images, that is, each element containing N1 ×N2 pixels can be represented as N1 ×N2 ×1 dimensional vector, which are denoted as vectors Xpq and Ypq , respectively. Obviously, the angle θ between Xpq and Ypq or vector distance between them, determines the degree of similarity between them. The similarity between these two subgraphs can be described by either the angle θ or the vector distance between them. When the distance between image vectors Xpq and Ypq is used to express similarity, it is called the minimum distance metric. This distance can be represented by the norm difference of the vector Xpq and Ypq ε  Xpq − Ypq

(2.105)

The definition of vector difference, distance similarity between image vectors Xpq and Ypq can be used by any kind of similarity algorithm in formulas (2.8)–(2.21) to judge, can also be used any kind of acceleration algorithm in formulas (2.23)–(2.28) to achieve. For example, the absolute difference measure (AD metric) is used. According to the abstract norm of ε, the AD metric can be defined as * * D(p, q)  ε  *Xpq − Ypq *

(2.106)

where D(p, q) represents the measure on position (p, q). The abstract norm of a vector is equal to the sum of the absolute values of its elements, so D(p, q) can be further represented as D(p, q) 

N1  N2  $ $ $Xp+i,q+j − Yp+i,q+j $ i

(2.107)

j1

For any obtained a D(p, q) value is compared with the threshold value k, if D(p, q) > 21 k, that difference before and after the template is larger, can determine the current area is part of a movement area, and marked the point, such as shown in Fig. 2.28a. In the process of tagging, you can compare and find out at the same time, the upper, lower, left, and right boundary points of the moving region are obtained by a rectangular moving block, as shown in Fig. 2.28b, i.e., the location of the callout points after the move, Fig. 2.28c shows a rectangular moving block created by moving the callout point.

2.4.4 Advancement in Real Time Algorithm In recognition of moving objects in front of the vehicle, due to each frame image needs OTSU threshold and two image similarity calculation, the calculation period of vehicle vision system will be short.

68

2 Vehicle Driving Safety Technology Based on IVP

(a)

(b)

(c)

Fig. 2.28 Rectangular moving blocks formed by callout points. a The motion of the callout point at the previous moment. b The moving position of the callout point at the following moment. c Rectangular moving block

Algorithms can become impractical if they can not be completed within the specified time (such as 100). There are many ways to increase the time of identifying an operation, such as: Set the pixel value of a scan line to n bytes, and the size of the bitmap array is n × bmHeight bytes. At this time, the 0 ∼ n − 1 bytes of the bitmap array record the pixel values of the first scan line of the image, the n ∼ 2n − 1 byte records the pixel values of the second scan lines of the image. And so on, the (j − 1)×n ∼ j ×(n − 1) bytes of the bitmap array record the pixel values in line j of the image. The n bytes of line j are denoted as b0 , b1 , …, bn−1 , respectively, as the 24 bit image, when bmBitsPixel  24, the b0 (blue), b1 (green) and b2 (red) records the first pixel value of the scan line j of the bitmap, b3 ,b4 and b5 records the second pixel values in line j, and so on. With the information about the BMP image format, you can quickly read the gray values of each pixel in the corresponding position of the image array stored in memory. The flow of the calculation program is shown in Fig. 2.29. After the improved algorithm, the time needed to process an image can be no more than 30 ms, which can meet the requirements of real-time processing. Take the vehicle at a speed 60 km/h at the time of finding pedestrians, visual detection is that the pedestrians in front of the vehicle through the road to 10.5 m, after using the 30 ms time deal with an image, the vehicle is on 0.5 m. After system operation, the vehicle begins to decelerate and brake, and can stop before the zebra crossing, allowing pedestrians to pass safely through the road. By the way, when multiple moving objects enter the field of view, the system can use image segmentation, matching and tracking algorithms to learn the current scene information of multiple pedestrians who want to cross the road ahead. Therefore, it is possible to determine the control countermeasures that should be taken during the vehicle driving process based on the information.

2.5 Intelligent Rear View Technology of Automobile

69

Begin

Read the image width, height, and template size

Read in two consecutive frames bmp1 and bmp2

Seek threshold

Starting from the pixel point of the i = 1 and j = 1 , match the corresponding positions of the two frames

Do not callout this point

Move to the next point

No

No

Match?

Yes

Callout this point

Match finished? Yes

Find the center of the callout area, and mark it with cross

Replace bmp1 with bmp2, Read into a new frame as bmp2

Fig. 2.29 Real time improvement algorithm flow

2.5 Intelligent Rear View Technology of Automobile As everyone knows, in the process of moving vehicles, when ready to change lanes or turn left or turn right, have to rely on the driver to observe and judge the other party before and after the running condition of the vehicle to determine whether the driver’s own vehicle can perform a lane change, or turn left or turn right. At this point, the driver’s field of view is often blocked, or because of road complexity and other factors, is changing lanes, or turn left or turn right, as a result, the collision between the vehicle and the vehicle behind it is causing traffic accidents. In addition, restricted by road construction condition, the motor vehicles and non motorized lanes of many roads in cities can not be isolated from objects such as railings. At this time, the motor vehicle temporary parking to open the door the moment, because of negligence by personnel, car doors are prone to collisions with non motor vehicles or pedestrians.

70

2 Vehicle Driving Safety Technology Based on IVP

Although the road traffic safety law and its implementing regulations are emphasized “roadside parking should be close to the right side of the road, door switch shall not impede other vehicles and pedestrians, but because of the personnel on the car or rush to get off or lack of awareness, often result in the door opening process of accident injuries to pedestrians or other vehicles. What’s more, taxi drivers are eager for quick success and instant success. They often rush up and down passengers on any part of the road, resulting in heavy traffic accidents that hurt pedestrians and other vehicles. This is the current traffic situation, relying solely on the emphasis of “road traffic safety law” and its implementing regulations is clearly not enough, it is impossible to achieve to enhance the legal awareness and safety awareness to the people in the short term, therefore, it is also necessary to increase the superior performance of vehicles and prevent such road accidents through advanced technical means. The intelligent rear view technology of automobile has come into being in view of the above reality.

2.5.1 System Working Principle In automotive intelligent rear view system, the image acquisition process is realized by on-board pinhole cameras 6, 7 and 8 (see Fig. 2.1). Among them, the pinhole camera 6 and 7 assume the right and left rear view images of the vehicle respectively, and the pinhole camera 8 is responsible for the collection of the scene image at the rear of the vehicle. A smart car rearview system device, in addition to the 6, 7, 8 pinhole camera, including: driving state encoder, signal processor, channel decoder, electronic switch, lock device, voice prompting device and liquid crystal display. Wherein, the output interfaces of the three pinhole cameras 6, 7 and 8 are respectively connected with three corresponding image input channels of the signal processor. The driving state encoder output interface is connected to the digital signal input interface of the signal processor. The driving state encoder signal input interface is connected with the left turn, right turn and reverse signal output interface of the vehicle. The first output interface of the signal processor is connected with the input interface of the channel decoder. The signal processor second outputs the interface to the input interface of the LCD. The signal processor third output interface is connected to the input interface of the voice player. The channel decoder output interface is connected to the control input interface of the electronic switch, and the output interface of the electronic switch is connected to the input interface of the lock (see Fig. 2.30). The driving state encoder includes: signal input interface, signal encoder and digital signal output interface. Wherein, the signal input interface comprises three input channels. The input of the first channel is connected to the left turn voltage signal output interface of the vehicle. The input of the second channel relates to the right turn voltage signal output interface of the vehicle. While, the input of the third channel is connected with the reversing voltage signal output interface of the vehicle. The signal encoder contains three monostable triggers, the input ports are

2.5 Intelligent Rear View Technology of Automobile

71

Fig. 2.30 Schematic diagram of automotive intelligent rearview system. 6 Right pinhole camera, 7 Left pinhole camera, 8 Rear pinhole camera, 9 Signal processor, 10 Channel decoder , 11 LCD, 12 Voice prompt, 13 Driving state encoder, 14 Electronic switch, 15–19 Lock device

respectively connected with the output ports of the three input channels of the signal input interface, the output port is connected with an input port of the digital signal output interface, the output port of the digital signal output interface is connected with the signal processor. The monostable trigger, as soon as the DC positive voltage is received, it is triggered, output high level “1” means “left turn” or “right turn” or “backward vehicle”, and there is switch voltage output. Otherwise, the output is low, “0”, which means “left turn” or “right turn” or “backward vehicle”, and there is no switching voltage output. The input/ output encoding of the driving status encoder is shown in Table 2.1. The output code is the digital code of the switch signal that represents the driving operation. A total of five lock devices are provided, the first lock device 15 is a left turn locking device, second lock device 16 is right turn lock, third lock 17 is left door lock, fourth lock 18 is the right door, and fifth lock device 19 is a reverse locking device.

Table 2.1 Driver status encoder output encoding table Left turn Right turn Backward vehicle 1 0 0 1 0 1 0 1 0

Output encoding

Driving state

100 101 010

Front left turn Left astern Front right turn

0

1

1

011

Right astern

0

0

0

000

Straight travel

0

0

1

001

Straight astern

72

2 Vehicle Driving Safety Technology Based on IVP

Each lock device includes an input interface, a relay switch and an electromagnetic decoupling device. As long as the intelligent vision system of the vehicle is in the working state, the images collected by the pinhole cameras 6, 7 and 8 will be sent to the signal processor in real time. The operation of signal processor for each image received by the image processing program module, can accurately identify whether there may be monitoring of traffic accidents caused by vehicles or pedestrians in the scene, so can lock in real time operation prohibited dangerous or driver error, or by other control devices, adopt avoidance, braking or speed control measures to automatically and effectively prevent possible accidents. As shown in Fig. 2.31, the image processing program module in the signal processor is the core module of the signal processor. The technology comprises an image preprocessing module, an image edge detection module, a decision calculation module and a decision execution instruction output module. The 4 input channels of the image processing program module are respectively connected with the output ports of the 3 analog-to-digital conversion modules and the output port of the digital signal input interface. The output port of the image preprocessing module is connected with the input port of the edge detection module of the image. The output port of the image edge detection module is connected with the input port of the decision arithmetic module. The output port of the decision arithmetic module is connected with the input port of the decision executing instruction output module. The first output port of the instruction execution output module is the first output port of the image processing program module and is connected with the channel strobe instruction output module. The second output port of the instruction execution output module is used as the second output port of the image processing program module and is connected with the input port of the output interface of the signal processor. The image preprocessing module receives the left, right and rear views of the left, right and back pinhole cameras. These images are transmitted to the image edge detection module after preprocessing. The image edge detection module detects the edge of the digital image, and outputs the two-valued edge image to the decision operation module. After transforming the two-valued edge images through polar logarithmic coordinates, the decision operation module uses the rotation invariance and scaling invariance of the image data conversion process to track the target, and then identifies the motion state of the captured object corresponding to the image and transmit it to the decision execution instruction output module. The decision instruction module determines whether a decision instruction is issued to follow up warning module and electromechanical servo device, according to the output code of the decision operation module and the switching signal of the driving operation. The output code of the decision operation module represents the motion state of the captured object. Decision making instructions include: prohibiting the left turn of vehicles, prohibiting the right turn of vehicles, prohibiting the opening of the left door of vehicles, prohibiting the opening of the right door of vehicles, and prohibiting vehicle reversing. Such as: 001 is to prohibit the left turn, 010 is to prohibit the right turn, 011 is to prohibit the opening of the left door, 100 is to prohibit the opening of the right door, and the 000 is to prohibit reversing the car.

2.5 Intelligent Rear View Technology of Automobile

Left view

Image edge detection module

73

Right view

Back view

Image preprocessing module

Decision arithmetic module

Decision instruction output module

Digital speech signal output interface

Driving operation switch signal

Channel gating instruction output module

Digital display signal output interface Fig. 2.31 Structure diagram of image processing program module

As shown in Fig. 2.31, the system indicates that the image preprocessing module is used to receive left, right, and rear views of the left, right, and rear pinhole cameras, after preprocessing, a digital image signal is transmitted to an image edge detection module. The image edge detection module detects the edge of the image, and outputs the two-valued edge image to the decision arithmetic module. The decision module will binarization edge image data conversion through polar log coordinate transform, using the image data conversion process with object rotation invariance and invariance to the zoom tracking of the target, and then identify the corresponding image captured by the object motion, and transmitted to the decision execution command output module. The decision execution module according to the output decision module receives the output, representing the state of motion capture of the object code, and the driving operation switch signal to determine whether a prohibited lane change, or turn, or open the door, or reversing the decision to execute instructions warning module and servo follow-up device. A liquid crystal display (LCD) is used to display the digital image signal output by the signal processor and the identification information of the scene image.

2.5.2 Intelligent Rear View Kernel Algorithm of Vehicle Intelligent rear view kernel algorithm (technical method), which includes two parts: image processing and scene state recognition. Among them, the edge detection of

74

2 Vehicle Driving Safety Technology Based on IVP

image processing uses mathematical morphology method, which is a commonly used algorithm for image edge detection.

2.5.2.1

Image Processing Method

Image processing includes four steps: image preprocessing, image edge detection, raster sampling and image conversion. 1. Image preprocessing The scene images of the left and right sides of the vehicle are collected in real time by two pinhole cameras mounted on the left and right rearview mirrors of the vehicle, and are transmitted to the on-board signal processor. The signal processor isolates the image from the color, the image is transformed into gray image and processed by equalization. 2. Image edge detection To preprocess the image, the edge detection is carried out based on the multi structure element mathematical morphology method: (1) Select the multi structuring elements according to the formula (2.70), so that the image center point is the coordinate origin, select N  3, so you can get 4 square structure element matrices of the 3 × 3 corresponding to θ  i × α  0◦ , 45◦ , 90◦ , 135◦ , ⎡

⎤ ⎡ ⎤ ⎡ ⎤ ⎡ ⎤ ◦◦◦ ◦◦• ◦•◦ •◦◦ B0  ⎣ • • • ⎦; B1  ⎣ ◦ • ◦ ⎦; B2  ⎣ ◦ • ◦ ⎦; B3  ⎣ ◦ • ◦ ⎦ ◦◦◦ •◦◦ ◦•◦ ◦◦• where • represents the active component in the structuring element, and ◦ represents for the invalid component in the structuring element. (2) Transform the data of the image, and use the structural elements of 4 square matrices B0 , B1 , B2 , B3 with 3 × 3 to do the gray dilation operation f ⊕ B on the input image f  f (x, y), and obtain 4 gray scale expansion results f ⊕ B0 , f ⊕ B1 , f ⊕ B2 and f ⊕ B3 are obtained. Then the B structure elements are used to perform grayscale corrosion operations on input images f , that is, the grayscale corrosion operations f B of the input image f are performed by using 4 structural elements of square matrices B0 , B1 , B2 , B3 with 3 × 3, 4 gray scale corrosion results f B0 , f B1 , f B2 and f B3 are obtained. The morphological gradient gi (f ) is obtained by combining the above expansion and corrosion calculation results. That is to say, 4 expressions g0 (f ), g1 (f ), g2 (f ) and g3 (f ) of morphological gradient gi (f ) are obtained by combining the results of expansion and corrosion.

2.5 Intelligent Rear View Technology of Automobile

75

(3) To do weighted synthesis of morphological gradient gi (f ), take ω1  ω3  0.4, ω2  ω4  0.1, and finally get the gray edge image. (4) The two-valued processing based on the principle of statistics is applied to the synthetic morphological gradient g(f ˆ ). In case of not opening the light of the car, take n  1, in case of turning on the light, take n  2, the gray mean variance of the pixel region eA is compared with the average gray level of the gray image frame one by one, and the pixel region is taken 1 or 0, and finally two-valued edge images are obtained. 3. Raster sampling To form a sampling circle region, the edge image of sampling circle area is sampled by polar index grid, That is, take the coordinate origin (x0 , y0 ) as the center of the circle and half of the image frame width as the sampling maximum radius rmax to form a sampling circle area, as shown in Fig. 2.32. 4. Image conversion The edge images obtained by polar grid sampling are converted into image data by the method of polar logarithmic coordinate transformation, its transformation formula is ⎧ " ! ⎪ ⎨ u  ku ln r rmin (2.108)   ⎪ ⎩ v  θ  kv arctan y x Among them, ku and kv are the constants of range resolution and angle resolution, and are set by experiment+according to the actual situation as ku  40.23, kv  20.4, rmin  10 pixels. r  x2 + y2 is the length of the pixel from the center of the transform, (x, y) is the coordinates of the image in Cartesian coordinates, (u, v) is

Fig. 2.32 Polar exponential grid and polar logarithmic coordinate transformation

76

2 Vehicle Driving Safety Technology Based on IVP

the coordinate in the polar log grid coordinate system obtained after transformation, u is the abscissa axis of the polar logarithmic grid coordinate system, and the length of its corresponding pixel distance is the center of the transform, v is the ordinate axis in the polar logarithmic grid coordinate system, equal to the angle between the polar diameter of the pixel and the horizontal axis, rmin is the minimum radius of sampling when polar coordinate transformation is performed.

2.5.2.2

Motion State Recognition

Motion state recognition is carried out according to the image data after conversion. That is to say, according to the edge image, the coordinate set Q  {ui |i  1, 2, 3, . . . ; ui ∈ U } is projected on the abscissa of the polar logarithmic coordinate system to analyze the moving state of the object behind the vehicle. Among them, U is the domain of the coordinate ui of the polar logarithmic coordinate system. Specific algorithms are as follows. Using the coordinate transformation from Cartesian coordinates to pole logarithmic grids, object tracking is performed with object rotation invariance and scale invariance. Select ui , take ui (k) as the sampling value of ui at the time t. When the sampling period is t, the ui (k + 1) represents the sampling value of ui at the next sampling time t + t. For example, select ub , and use ub (k) to express the sampling value of ub at the time t. After the sampling period is t, the ub (k + 1) represents the sampling value of ub at the next sampling time of t + t. Determine the size relationship between ui (k) and ui (k + 1), if |ui (k + 1)| < |ui (k)|, that indicates the selected ui represents the stationary object behind its own vehicle, or that its speed is lower than its own vehicle. Otherwise, the object corresponding to the selected E is a relatively high velocity object whose motion is faster than its own vehicle. Through the operation of the above step, the signal processor can make decision according to the result of the recognition of the movement state and output the execution instruction. For example, |ui (k + 1)| > |ui (k)| appears in the left view, indicating the speeding of the rear left side of the car. At this time, if the driver switches the left turn indicator switch, signal processor will output real-time control instructions to the driver by voice, lights and warning screen, at the same time through the electrical device to prevent the driver left or left lane changing operation. Similarly, when the |ui (k + 1)| > |ui (k)| appears in the right view, if the driver switches the right turn indicator switch, signal processor can output real-time control instructions to the driver by voice, lights and warning screen, at the same time through the electrical device to prevent the driver’s right or right lane change operation. When the vehicle is in a suspended state, |ui (k + 1)| > |ui (k)| appears in the left view, the signal processor will output real-time control instructions to the driver by voice, lights and warning screen. At the same time, the door on the left side of the car is locked by an electromechanical device, and the operator is not allowed to operate the left door. |ui (k + 1)| > |ui (k)| appears in the right view, control command output

2.5 Intelligent Rear View Technology of Automobile Left view

Image edge detection

Recognition operation

77

Right view

Image preprocessing

Driving operation switch signal

Decision instruction output

Peripheral equipment

Fig. 2.33 Vehicle intelligent rear view algorithm flow

module can output real-time control instructions to the driver by voice, lights and warning screen, and an electromechanical device locking on the right side of the vehicle door, prohibit passengers on the right side of the door operation. The algorithm flow is shown in Fig. 2.33. Compared with the ordinary car, the car has an intelligent rear view function, which can achieve obvious technical results. That is to say, the level of automation and intelligence has been further improved in ensuring the safety of vehicle driving, embodied in: (1) Not because of the driver’s negligence in the lane change process, or left turn, or right turn, with the left rear or right rear vehicle collision. (2) When the vehicle is stopped, it will not, because the driver is negligent and is opening the door in the process of collision with the motor vehicle, non motor vehicle or pedestrian in the rear. (3) When reversing the vehicle, the road condition behind the driver can be reminded in time, so that the operation safety of the existing vehicle and the life safety level of the vehicle can be improved.

2.6 Automatic Identification Technique for Traffic Signs Traffic signs mainly include speed limit sign and direction indicator.

2.6.1 Automatic Identification of Road Speed Limit Signs For the safety of road and vehicle driving, urban roads and high-grade highways, especially motorway, traffic agencies have to set speed limit signs at each road section to remind the driver to control the speed, drive carefully, avoid illegal or even cause

78

2 Vehicle Driving Safety Technology Based on IVP

traffic accidents because of the speeding of vehicles. However, when the current vehicle is scarce (traffic flow is small), vehicle drivers often neglect their driving speed and overspeed. Although with the progress of science and technology, many roads have installed the vehicle speed monitoring device, but from its effect, it is basically a punitive technical means and measures, often after the speeding, the driver must be warned to comply with the traffic regulations by means of punishment. The existing road speed monitoring technology can not send warning messages to vehicle drivers in real time. Obviously, this is not a humanized technology.

2.6.1.1

System Hardware Foundation

The hardware foundation of the automatic identification system of road speed limit signs includes: three pinhole cameras, signal processors, speed control mechanisms and voice prompts. The three pinhole cameras are shown in Fig. 2.1, where 1, 3 and 5 (vehicles in accordance with the right driving rules), or, 2, 4, and 5 (vehicles in accordance with the left driving rules). The output interfaces of the three pinhole cameras are respectively connected with three corresponding image input channels of the signal processor, the output interface of the signal processor is connected in parallel with the input interface of the speed control mechanism and the input interface of the voice prompting device. The three pinhole cameras transmit analog signals of the speed limit sign information collected by each of them to the signal processor through their respective output interface and video cable respectively. In the 3 pinhole cameras, the pinhole camera 1 (or 2) is used to collect speed limit sign images on the road ahead of the vehicle. The sign usually uses white or yellow paint to spray speed limits on the driveway, the driver controls the speed of the vehicle on the front lane by observing the speed limit on the road ahead of the vehicle. For example, on the urban elevated road, the speed limit number “60–80” will be sprayed with white paint at the specific intersection or road section, which means that the driver must control the speed between 60 and 80 km/h on this section. Pinhole camera 3 (or 4) for the acquisition of the right side of the road (or the left side of the road) speed limit sign image, vehicle speed is set on the right side of the road signs have been using electronic devices, according to the weather condition and real-time adjustment of digital sign. For example, on the freeway, the speed limit mark of LED (light emitting diode) will be set on the right side of the vehicle direction. It can adjust the speed limit of each section of the road from the road monitoring center according to the climate change. On sunny days, it will display “120”, which means that the speed limit of vehicles must be 120 km/h. When the weather is foggy, it will display “60”, which means that the speed limit of the vehicle must be 60 km/h. The pinhole camera 5 is used to collect the speed limit sign images set on the top of the road, the vehicle speed limit sign commonly used fluorescent color image identification, spraying white circles on the dark green floor, white spray speed digital in the white circle. For example, when the number in the logo circle is “80”, it indicates that the speed of the road segment should not exceed 80 km/h.

2.6 Automatic Identification Technique for Traffic Signs

79

The signal processor module is used to automatically identify the operation of the speed limit sign on the road, as shown in Fig. 2.34. It includes: three image input channels corresponding to the camera 1, 3, 5 (or 2, 4, 5) and their analog digital conversion module, voltage signal module/digital conversion module, image processing module, output interface and memory module. Wherein, the input terminal of the voltage signal analog to digital conversion module is connected with the output interface of the vehicle speed sensor, and the output end of the module is connected with the second input port of the decision instruction output module in the image processing module. The output port of the image processing module is connected with the input end of the output interface. The template output recording port of the image processing module is connected with the database writing port of the memory module, and the template reading port of the image processing module is connected with the database reading port of the memory module. The image processing module is the core module of the signal processor, as shown in Fig. 2.35. It includes: image preprocessing sub module, speed limit identification sub module and decision instruction output sub module. Among them, the three input channels of the image preprocessing sub module are respectively connected with the output terminals of the three image analog-to-

Vehicle speed control mechanism

Image A/D module 1

Analog image input channel 2

Image A/D module 2

Analog image input channel 3

Image A/D module 3 Voltage signal A/D module

Image processing module

Analog image input channel 1

Voice player

Output interface

Memory module

Simulated speed voltage signal

Fig. 2.34 Schematic diagram of internal structure of signal processor

Image preprocessing sub module

Speed limit identification sub module

Digital image signal Image processing module

Simulated speed voltage signal

Decision instruction output sub module

Voltage signal A/D module

Fig. 2.35 Schematic diagram of image processing program module

80

2 Vehicle Driving Safety Technology Based on IVP

digital conversion modules. The output port of the image preprocessing sub module is connected with the input port of the speed limit identification sub module. The output port of the speed limit identification sub module is connected with the first input port of the decision instruction output sub module. The template output record port of the speed limit recognition sub module is the template output record port of the image processing module. The template reading port of the speed limit recognition sub module is the template reading port of the image processing module. The second input port of the decision instruction output submodule is connected with the output port of the voltage signal analog to digital conversion module, the output port of the decision instruction output module is used as the output port of the image processing module and is connected with the input end of the output interface of the signal processor. The image preprocessing sub module receives three speed limit sign images transmitted by three pinhole cameras, after preprocessing, the digital image signals are transmitted to the speed limit identification sub module. Speed limit identification sub module, when the matching template is collected, the template of the real speed road sign matching template is written into the speed limit sign of the memory module to match the template database through the template output record port. When the speed limit identification is implemented, the read port of the template reads the matching template in the memory module which is placed on the highway speed limit sign matching template database. The identification algorithm of highway speed limit sign is done by the method of normalized rotation inertia of the image geometry, that is, NMI (Normalized Moment of Inertia) method. Finally, the identification results are output in binary code to the output module of the decision instruction. Decision command output module, according to the output signal of the digital information output speed recognition from the first input port receives the module and vehicle speed sensor from the input port second to receive the digital information after the analog-to-digital conversion, thereby generating a control command. The control command is output to the input interface of the speed control mechanism and the voice prompt through the output interface of the signal processor. Under the influence of the control command, the speed is automatically controlled within the recognized speed limit range, while the voice prompt broadcasts the speed limit information at the same time. The data structure of decision instruction consists of two 8-bit-bytes. The first 8-bit-byte represents the speed limit sign recognition value, for example, “01111000” indicates that the speed limit value is “120 km/h”. The highest level (1 bits) of second 8-bit-byte indicate whether there is an overspeed, and the 7 bits on its right side indicates the speed of the vehicle, such as: “10110010” represents the current vehicle already exceeding the speed of 50%. The so-called normalized moment of inertia (NMI), that is, the concept of physical rotation inertia is used to define the centroid of the gray image and the moment of inertia around the center of mass of the image, and then the moment of inertia of the image around the center of mass is given. NMI has scaling, rotation and translation invariant properties, so it can be used as an object recognition feature.

2.6 Automatic Identification Technique for Traffic Signs

81

The workflow of the image processing module is divided into three steps. Firstly, the highway speed limit sign image is pre processed so as to enhance the image, separate the background and transform it into gray image. Secondly, the matching template is called in order to implement the recognition of the highway speed limit sign. Finally, the recognition results are converted to control instruction output. The speed control mechanism is shown in Fig. 2.36, which comprises a first digital to analog converter, a rectifying diode, a power amplifier, an electromagnetic valve and an electric brake push rod. Wherein, the input port of the first digital to analog converter is the input interface of the speed control mechanism, the output terminal of the first digital to analog converter is connected with the positive electrode of the rectifying diode, the negative electrode of the rectifier diode is connected with the positive point of the input port of the power amplifier, the output port of the power amplifier is connected with the positive end of the electromagnetic valve coil and the positive end of the electric brake coil of the electric brake push rod. The second output end of the first digital to analog converter is connected with the ground terminal of the power amplifier, the negative pole end of the solenoid valve coil and the negative end point of the electric brake push rod coil. The electromagnetic valve coil is outside the electromagnetic iron core. When the electromagnetic valve coil is charged, the magnetic attraction is generated through the electromagnet core. With the change of the point voltage at the two ends of the solenoid valve coil, the magnetic attraction of the electromagnetic iron core changes accordingly. The magnetic attraction of the electromagnetic iron core acts on the valve and pulls the valve to change the opening size of the valve. A resistance spring is a tension spring. When the valve is pulled by the magnetic attraction of the electromagnetic iron core, the resistance spring is also stretched at the same time, thereby producing an elastic force opposite to the magnetic attraction of the electromagnetic iron core. When the magnetic attraction of the electromagnetic iron core and the resistance spring force balance, the valve is stopped pulling. In other words, the valve stops at one opening, which corresponds to the two terminal voltage of the solenoid valve coil. The electric brake push rod comprises an electric brake coil and an electromagnetic push rod. The electric brake coil is sheathed at one end of the electromagnetic push rod, and the other end of the electromagnetic push rod is connected with the pedal brake rod mechanism. When the electric brake coil is powered by electricity, the electromagnetic field generated by the electric brake coil generates an axial mechanical thrust on the electromagnetic push rod arranged in the electric brake coil. The axial thrust acting on the mechanical pedal brake lever mechanism for electric brake push rod force, through the lever mechanism and a pedal brake to the same effect of the vehicle automatic braking. The magnitude of the braking force is directly controlled by the voltage value of the two end points of the electric actuator brake rod coil. The working principle of the speed control mechanism is as follows. When the vehicle is in normal operation, the solenoid valve is in full open state, the valve opening size is 100%, and the electric brake push rod does not exert braking force on the brake rod. The control command decision command output module shows that the vehicle is in overdrive state, control instruction by the input interface speed control mechanism of the transmission to the first DAC digital control instruction is

82

2 Vehicle Driving Safety Technology Based on IVP

Control command digital signal input

Fig. 2.36 Schematic diagram of speed control mechanism. 20 First DAC, 21 Rectifier diode, 22 Power amplifier, 23 Solenoid valve, 24 Electric brake push rod, 25 Solenoid valve coil, 26 Electromagnetic core, 27 Valve, 28 Drag spring, 29 Valve body, 30 Electric brake coil, 31 Electromagnetic push rod

converted to analog output. The analog voltage is a positive voltage signal. Through the forward conduction of the rectifying diode, the output voltage signal of the digital to analog converter is sent to the input port of the power amplifier. The rectifier diode is connected between the output terminal of the first digital to analog converter and the input port of the power amplifier. The power amplifier amplifies the received voltage signal and outputs the power signal, at the same time, it is added to the two voltage signal input endpoints of the electromagnetic valve coil and electric brake push rod coil each, reducing the solenoid valve opening size, so that the vehicle to form a braking force. The braking force is transmitted by the torque of the bar mechanism to drive the foot brake and force the vehicle to decelerate gradually. The reduction of valve opening and the increase of brake force depend on the voltage signal on both coils. The opening of the solenoid valve decreases with the increase of the voltage, the braking force increases with the increase of the voltage. In contrast, the control instruction output by the decision instruction output module indicates that the current vehicle is not in overdrive, the analog voltage signal generated by the digital control command after the first digital converter is negative, therefore, under the action of a rectifying diode, the negative electrode of the diode is not output voltage signal, the power amplifier has no power output, the system does not automatically on the solenoid valve coil and electric brake push rod coil voltage signal, no vehicle driving intervention effect. The working process and principle of the automatic identification system for road speed limit signs are described as follows. Three pinhole cameras mounted on the vehicle collect the speed limit signs of the vehicle in the three directions in front of the vehicle. The three pinhole cameras respectively transmit the analog signals of the road speed limit signs collected from each other to the signal processor through their video cables respectively. The signal processor of three images of pinhole camera to collect real-time processing, in order

2.6 Automatic Identification Technique for Traffic Signs

83

to identify the current image on the section of the vehicle speed limit signs, and will speed the recognition results by the output control command. The control command is transmitted to the vehicle speed control mechanism through an output interface of the signal processor. When the speed is greater than the speed limit identification value, the speed control mechanism can reduce the vehicle fuel flow and reduce the speed in real time, and warn the driver that there is overspeed at present, otherwise, the device will not cause any adverse effects or interference on the vehicle.

2.6.1.2

Automatic Identification Algorithm of Road Speed Limit Signs

The module connection of the automatic identification system for road speed limit signs is shown in Fig. 2.37. The road speed limit sign process, automatic identification includes the following steps: establishing a template database, speed limit sign image online acquisition, image enhancement processing, mark area search, template matching, control command output etc. 1. Build template database The template database consists of three categories, the first type of database, for storing speed signs, image templates that are set at the top of the road ahead of the vehicle. Second type of database used to store speed limit image templates on the road ahead of the vehicle. The third type database is used to store the speed limit sign image template set on the right side of the road. The urban road speed limit sign stored in the first type of database is characterized by the Arabia digital face up imaging with circle. The image of speed limit sign stored in second type database is characterized by Arabia digital positive squint imaging of coarse font.

First pinhole camera

First image input interface

Second pinhole camera

Third pinhole camera

Second image input interface

Third image input interface

Signal processor

First database

Second database

Third database

Voice prompt

Vehicle speed control mechanism

Fig. 2.37 Module connection for automatic identification system of speed limit sign

84

2 Vehicle Driving Safety Technology Based on IVP

The speed limit sign image template stored in the third type database is characterized by the digital lateral squint imaging with circle LED display. Each type of database has the same database structure, the database structure contains, speed limit sign image template, template header code, speed limit value, template matching rate, and so on field variables, namely each entries in a database are included: , , , four variable data field. The template of the three databases adopts the method of real shot to collect and extract the speed limit sign image. The technique is designed to allow experimental vehicles to be collected at certain distances along urban roads, urban viaducts, and motorways, where all existing speed limit signs are located. The so-called “fixed distance” refers to the camera in the acquisition speed limit signs should be able to grasp the appropriate shooting distance, among them, 50 m is equipped with a telephoto view of pinhole camera to collect the speed limit signs the best image shooting distance. 2. Online collection of speed limit signs The three pinhole cameras collect images of speed limit signs in each field of view, and each captured image is transmitted to the signal processor along the corresponding three image transmission channels. The timing acquisition of the invention means that the signal processor determines the time interval t between the three pinhole cameras at each time the image is collected according to the current vehicle speed, and t 

L u

(2.109)

In the model, u is the current vehicle speed, and the unit is m/s, L is the image sampling interval, unit is m. Such as, take L  10 m, when u  120 km/h, t  0.3 s, that is, each interval 300 ms to acquire an image. Take L  10 m, when u  30 km/h, t  1.2 s, that is, an image is acquired at each interval 1.2 s. Each image collected is transmitted to the signal processor along the corresponding three image transmission channels. 3. Image enhancement Firstly, the image is enhanced and the noise is removed, and then stored in the buffer. Each buffer area stores three images captured by three pinhole cameras, each of which has its own frame head code, and the frame header code represents the corresponding pinhole camera and its corresponding type database. For example, the frame header code “01”, “10”, “11” means the images collected from the first, second and third of the pinhole camera, also said the images collected will be respectively matching calculation template with the first, second and third types of data in the speed limit sign.

2.6 Automatic Identification Technique for Traffic Signs

85

4. Sign region search For the images from different transmission channels, the corresponding speed limit search area is determined, and the neighborhood of the speed limit sign is used as the template to match the image to identify the main search region. For example, the speed limit sign that may be included in the image from the first image transmission channel is a Arabia figure with a circle, the signal processor is to match and recognize the images acquired at about 50 m of the front of the vehicle, in this image, if there is the speed limit, Arabia limit is generally a circle of digital signs are in middle upper position, so the signal processor can be near the location as the main search area recognition operation. The view angle is 20°, telephoto lens 768 × 576 resolution CMOS camera as an example, consider necessary to prevent omission of redundant, the speed limit signs on image collected in the area around the center point is 192 × 144 pixel area, namely the center point around 192 × 144 pixel region will the template and the image is the main search area matching operation. 5. Template matching (1) According to the frame header code of the image, the speed limit sign template of the corresponding type database is retrieved, and the normalized rotation inertia method is used to perform the matching recognition operation. If the retrieved speed limit template matches the speed limit flag in the current image, go to step 6. Otherwise, the next speed limit template is called from the same type of database to continue matching recognition operations with the current image, if the match is not achieved until the template call is finished, there is no speed limit flag in the current image, the next frame head code image is reread, and the matching recognition operation of step 5 is repeated. Speed limit template, the record in the database contains template matching rate λij , defined as follows λij 

nij × 100% Ni

(2.110)

Among them, λij is the matching rate of the template j in type i database, Ni is the number of frames to be identified with a speed limit flag, which are collected by pinhole camera i. nij is the cumulative number of matches that are matched by the template j in type i database. λij will be continuously updated with the mileage of the vehicle, i  1, 2, 3, j  1, 2, . . .. In order to improve the efficiency of the matching recognition, each time the template is retrieved, it starts with a high matching rate template, followed by a sub high matching rate template, and so on, until all of the template calls are finished. The template with the same matching rate is called in permutation order, and the matching recognition operation is carried out only in the search area of the speed limit sign of the image, so that the time of the whole matching recognition operation can be shortened. The normalized moment of inertia (NMI) method described here, the centroid (Cx , Cy ) of the gray image f (x, y) is defined as follows

86

2 Vehicle Driving Safety Technology Based on IVP

M N x1

y1

Cx  M N Cy 

xf (x, y)

x1 y1 f (x, y) M N x1 y1 yf (x, y) M N x1 y1 f (x, y)

(2.111)

(2.112)

Among them, the centroid (Cx , Cy ) represents the gravity center of gray of the image. The moment of inertia of the image around the center (Cx , Cy ) of mass is recorded as J 

M  M  N N    2 (x, y) − (Cx , Cy ) f (x, y)  ((x − Cx )2 + (y − Cy )2 )f (x, y) x1 y1

x1 y1

(2.113) According to the definition of the center of mass and the moment of inertia of the image, the normalized rotation inertia of the gray image around the center (Cx , Cy ) of mass can be given, that is, NMI (Normalized Moment of Inertia)   √ M N 2 2 x1 y1 ((x − Cx ) + (y − Cy ) )f (x, y) J NMI   (2.114) M N m x1 y1 f (x, y)  N Among them, m  M x1 y1 f (x, y) is the image quality, representing the sum of all gray values of the image. The NMI has the invariant properties of scaling, rotation, and translation. This refers to the integral area in a given area, where summation is used instead of integration for digital images, there are 7 invariant moments, which are invariant to translation, rotation, and scale change. (2) Each template is successfully matched once, and the signal processor automatically matches its cumulative template rate, and updates the template in the database record field variables data. (3) According to the vehicle speed limit sign recognition of value is compared with the current vehicle speed, whether the vehicle “is speeding”, when the vehicle speed v exceeds the speed limit VM , which is v − VM > 0, we can calculate the degree of speeding α

v − VM × 100% VM

The control instruction output is generated according to v and α.

(2.115)

2.6 Automatic Identification Technique for Traffic Signs

87

6. Control command output Under the influence of the control command, the speed control mechanism automatically limits the speed of the vehicle within the speed limit, and warns the driver to master the speed limit information and drive carefully. 7. The process of repeating steps 2 through 6 The above technical steps are expressed in the flow chart, as shown in Fig. 2.38. The technology has the following beneficial effects, and further improves the level of automation and intelligence in ensuring the safety of vehicle driving. Concrete manifestation, First, not because of the driver’s negligence and the vehicle speeding phenomenon, the accuracy can reach 99.5%. Second, can significantly reduce road monitoring and management costs, saving economic resources. Third, it can significantly reduce traffic accidents caused by speeding. Fourth, it can significantly improve the driver’s awareness of traffic legal system, and enhance the harmonious atmosphere between traffic enforcement and law abiding.

Online speed limit sign image

Recognition operation No

Current template database No

Is this the last template? Yes

Yes

Other template databases

Recognition operation

Cumulative template matching rate

Match successfully?

Yes

Is it speeding?

No

Yes

No No

Match successfully?

Control instruction output Is the final template? Yes

Vehicle speed control mechanism

Fig. 2.38 Flow chart of identification of speed limit sign

Voice prompt

88

2 Vehicle Driving Safety Technology Based on IVP

2.6.2 Automatic Recognition of Directional Signs As shown in Fig. 2.39, in the case of current traffic management, the driving direction indicator generally comprises straight lines, turn left, turn right, go straight and turn left, turn left and turn right, straight and right turn, 6 types. Vehicles must follow the indicated lane. However, things often happen: due to driver neglect or other reasons (such as: the driver’s sight is blocked), often resulting in vehicles not driving in the lane, or vehicles at intersections waiting for traffic lights “stop at the wrong lane”. Therefore, the original driving route of the vehicle is “destroyed” and even affects the whole road traffic. When the vehicle has an automatic identification of the moving direction sign, the probability of such a “wrong lane” or “stop the wrong road” can be reduced to a minimum, so that the vehicle running efficiency can be improved. In order to realize the automatic identification of the moving direction sign, the automobile can also rely on the intelligent vision technology. The system hardware and its working process needed to realize this function are the same as those of the vehicle intelligent identification road obstacle and the pit system, but its core algorithms are not the same. When the vehicle signs the moving direction sign on the road, the signal processing and analysis information of the signal processor is from the image signals collected by the pinhole cameras 1 and 2. When the vehicle signs the moving directions indicated above the road, the signal processor processes and analyzes the information collected from the pinhole camera 5 (see Fig. 2.1). The intelligent vision system of a vehicle can indicate the lane of the vehicle in real time to the driver in real time through the recognition operation. When the lane indicating automatic identification system finds that the signal of the direction light does not correspond with the current lane of the vehicle, the system immediately provides warning information to the driver, thus avoiding possible traffic violations or accidents.

Fig. 2.39 Road traffic sign map

2.6 Automatic Identification Technique for Traffic Signs

2.6.2.1

89

Frequency Domain Feature Extraction Method

Considering the influence of illumination, mechanical friction and other factors on the direction of the road, there is a big difference between color and brightness, therefore, we must use the relation between affine transformation in space and frequency domain, and implement the feature extraction method for two-valued images in Fourier frequency domain, it is better to extract the middle and low frequency characteristics of the image, suppress high-frequency noise, and avoid the adverse effects of illumination changes on the moving direction indicator image analysis. The algorithm of frequency domain feature extraction is as follows. Firstly, noise is removed from the image in frequency domain. Secondly, edges are extracted from the image. Finally, the Fourier transform is implemented on the edge image, and the image features are extracted in frequency domain, after the feature extraction, the vector whose amplitude is the magnitude of the spectrum is the feature vector of the image. If the image is affine transformed, the features of the object in the image can be obtained under affine transformation. The core algorithm is frequency domain noise reduction and frequency domain feature extraction. 1. Frequency domain noise reduction Since the driving direction indicator image must contain a variety of noise, in order to increase the accuracy of the identification algorithm, the image denoising needs to be carried out, this is particularly important for low signal-to-noise ratio images. In view of the target recognition in uniform background, when the noise causes the change of pixels in the background region, the spatial filtering will lead to uneven distribution of background luminance, while the filtering effect in frequency domain is relatively stable, therefore, the frequency domain denoising method is applied to the moving direction indicator image. Suppose the original image is f (x, y), and the image after Fourier transformation is F(u, v), frequency domain enhancement is used to select the appropriate filter H (u, v) to process the spectrum components of F(u, v) and obtain the processed Fourier function G(u, v), that is G(u, v)  F(u, v) · H (u, v)

(2.116)

Then turn the G(u, v) through the Fourier inverse transform, get the enhanced image g(i, j). Because the noise is mainly concentrated in the high frequency part, in order to remove noise and improve image quality, low-pass filter H (u, v) is usually used to suppress high frequency components and pass low frequency components, then, the inverse Fourier transform is used to obtain the filtered image, so that the smooth image can be achieved. The low-pass filter H (u, v) in the frequency domain contains ideal low pass filter, Butterworth low-pass filter, exponential low pass filter and trapezoidal low-pass filter, etc. For traffic direction indicator images, frequency domain noise reduction can be performed using the Butterworth low-pass filter.

90

2 Vehicle Driving Safety Technology Based on IVP

Set the ideal low pass filter on the Fourier plane, the cutoff frequency from the origin is D0 , and the transfer function of the Butterworth filter with order n is H (u, v)  1+

!√

1 2−1

"

D(u,v) D0

2n

(2.117)

In the formula, D(u, v) is the distance from point (u, v) to frequency domain coordinate origin, that is D(u, v) 

√ u2 + v 2

(2.118)

As shown in Fig. 2.40, this is the two-valued image of the road traffic sign shown in Fig. 2.39 and the corresponding frequency domain image after the Fourier transformation.

Fig. 2.40 Two-valued images and their frequency domain images of traffic signs

2.6 Automatic Identification Technique for Traffic Signs

91

2. Affine invariance based on frequency domain features   Assuming that f x , y is the affine point of the original pixel f (x, y), then          b1 a11 a12 x x x +  A + b, |A|  0 (2.119) y y a a y 21 22 b2  Among them, A 

a11 a12 a21 a22



is the rotation matrix of affine transformation,  T in which the element is called rotation parameter, b  b1 b2 is the translation vector of affine transformation, in which the element is called translation parameter. Under the two-dimensional conditions in the image space domain, we can define the two-dimensional Fourier transform of the image f (x, y) as follows +∞ +∞ F(u, v)  F{f (x, y)} 

f (x, y) · e−j(ux+vy) dxdy

(2.120)

−∞ −∞

Its inverse transformation is as follows 1 f (x, y)  F {F(u, v)}  4π 2 −1

+∞ +∞ F(u, v) · ej(ux+vy) dud v

(2.121)

−∞ −∞

The image spectrum after affine transformation follows    G(u, v)  F f x , y +∞ +∞  f (x, y) · e−j(ux +vy ) dx dy

(2.122)

−∞ −∞

Focuses on the connection from formulas (2.119) to (2.122), the relation between the spectrum F(u, v) before affine transformation and the spectrum G(u, v) after affine transformation is as follows   a22 u − a21 v −a12 u + a11 v 1 j |A|1 [(a22 u−a21 v)b1 +(−a12 u+a11 v)b2 ] F G(u, v)  e , |A| |A| |A| (2.123) Among them, j2  −1. From the formula (2.121) visible, after the affine transformation, the spectrum of the image has similar characteristics to the spectrum of the original image. The difference is that the scale in the two directions of x and y changes. The important point is that the translation of the airspace is eliminated. This conclusion guarantees

92

2 Vehicle Driving Safety Technology Based on IVP

that the image is affine transformed, and then the feature is extracted in frequency domain, and the whole pose information of the target can be obtained, at the same time, the displacement between different images of the target under the same posture can be eliminated. 3. Frequency domain feature extraction See Fig. 2.41. The size of the spectrogram F(u, v) is N × N . The center of the frequency map is set as the coordinate origin, and the feature location point (u, v) is extracted to form the set S, there is   N N (2.124) S  {(uα , vα )}; α ∈ − , 2 2 And uα+1  2uα vα+1  2vα

(2.125)

That is to say, the coordinates of the positions extracted from the frequency domain show a geometric series distribution feature Because the Fourier conjugate spectrum is symmetric, so only half of the frequency domain graph for feature extraction, the spectral amplitude of each extraction point by from low frequency to high frequency sequence into one-dimensional vector, as frequency domain features of images.

Fig. 2.41 Feature location map for spectrum extraction

2.6 Automatic Identification Technique for Traffic Signs

2.6.2.2

93

Recognition Principle

The Principal Component Analysis (PCA) method is used for the identification of the lane sign. This is an optimal orthogonal transformation based on the statistical properties of the target. The basic idea of principal component analysis is to find the transformation method of main feature components according to the correlation of attributes in data sets. That is to say, the attributes of the dataset can vary from more than ten to hundreds, and as the means of information collection are developed, the attributes can even be up to thousands. Although each attribute of the dataset provides some information, the amount and importance of information provided are different. Moreover, in many cases, there are different degrees of correlation among attributes, so that the information provided by these attributes is bound to overlap. By means of PCA transform algorithm, we can find out the components that highlight the main features of the recognized image, which is called the principal component. Therefore, PCA can reduce the whole computation greatly and improve the recognition speed. Principal component analysis (PCA) is also called K-L transform (KarhunenLoéve Transformation). 1. K-L transform K-L transform is y  AT (x − μx )

(2.126)

The inverse K-L transform is expressed as x  Ay + μx

(2.127)

  Among them, the transformation matrix A  a1 a2 · · · an of order n×n satisfies (1) AT A  AAT  I is an orthogonal matrix, in which I is a unit matrix, and column vector ai satisfies the standard orthogonality, that is ai · aj  aiT aj 

1ij 0 i  j

(2.128)

  (2) AT  x A  , among them,   diag λ1 λ2 · · · λn , λ1 ≥ λ2 ≥ · · · ≥ λn is the eigenvalue of  x , and it is arranged from big to small. The column vector ai is a characteristic vector of  x , that is  x ai  λn ai Among them, the angle said the statistical characteristics of vector x.

(2.129)

94

2 Vehicle Driving Safety Technology Based on IVP

It is easy to obtain the mean vector and covariance matrix of the transformed y, that is μy  E{y}  E{A(x − μx )}  A(E{x} − μx )  0 (2.130) ,  T  T   E A (x − μx )(x − μx )T  AT  x A    y  E y − μy y − μ y (2.131) 2. K-L transform property (1) Minimize the “two multiplied” (square) error of dimension reduction estimation. The component yi  ai · (x − μx )  aiT (x − μx )(i  1, 2, · · · , n) in formula (2.127) is used to recover x, in order to obtain x

n 

yi ai + μx

(2.132)

i1

The so-called “two-multiplication” is the method of calculating the sum of squares of the distance between the observation point and the estimated point. The “twomultiplicative” refers to the use of “square” to measure the distance between the observation point and the estimated point (in ancient Chinese, it is called “square” as “two-multiplying”). The “two-multiplicative error minimization” is the operation of the least square method. Retain the component yi in 1 ≤ i ≤ m, and use some constant component ui instead of the yi correlation in the m < i ≤ n to estimate the x, the xˆ is estimated to be xˆ 

m 

yi ai +

i1

n 

ui ai + μx

(2.133)

(yi − ui )ai

(2.134)

im+1

The resulting error is x  x − xˆ 

n  im+1

Find the “two multiplied” (squared) error 

ε  E |x| 2

2







 E x x  E T

n  im+1

. (yi −

ui )2 aiT ai



n 

  E (yi − ui )2

im+1

(2.135) Minimize this error

2.6 Automatic Identification Technique for Traffic Signs

95

∂ε2  −2E{yi − ui }  −2E{yi } − ui  0 ∂ci

(2.136)

Because of E{yi }  0, so ui  0, the estimation formula (2.133) is simplified as xˆ 

m 

yi ai + μx

(2.137)

i1

y1 , y2 , …, ym is called the primary component, and the abandoned ym+1 , ym+2 , …, yn is called the non principal component. By using the ui  0 operation, the two multiplied error formula (2.135) is obtained ε  2



n  ik+1 n 

n ,   2 2 E yi  E aiT (x − μx ) ik+1 n   aiT E (x − μx )(x − μx )T ai  aiT  x ai



ik+1

(2.138)

ik+1

Let λ1 ≥ λ2 ≥ · · · ≥ λn be the eigenvalue of  x , and under the constraint of |a|  1, the Two-stage-type optimization can be seen, a that minimizes the level of aT  x a is equal to the eigenvector an , which corresponds to the minimum eigenvalue λn , it meets  x an  λn an . If further dimensionality reduction is required, then the n − 1 component is discarded. And so on, the base vector corresponding to the n − m components of the discard is the characteristic vector corresponding to the smaller n − m eigenvalues. Here ε2 

n 

aiT λi ai 

im+1

n 

λi

(2.139)

im+1

Take the minimum. It can be seen that after the K-L transform, the data recovered from the corresponding component of the small eigenvalue corresponding to the two multiplicative error is optimal, and the multiplicative error is the sum of the eigenvalues. According to this principle, a reduced dimension subspace is constructed, even though some components are removed, however, the original information can be restored with the remaining features, and the optimal two multiplied error level can be maintained. (2) Component variance maximization Consider the deviation of the vector x and the mean μx onto a shaft. The axis is represented by a vector a, its weight is y  a · (x − μx )  aT (x − μx )

(2.140)

96

2 Vehicle Driving Safety Technology Based on IVP

We hope that the new component after the projection should reflect the original information as much as possible, that is, the range of y change can be as large as possible, and the variance of y is used to reflect the range of change       vary  E (y − E{y})2  E y2  aT E (x − μx )(x − μx )T a  aT  x a (2.141) λ1 ≥ λ2 ≥ · · · ≥ λn is still assumed to be the eigenvalue of the  x . Under the constraint of |a|  1, the two-type optimization can be seen, a  a1 maximizes aT  x a, and a  a1 is the eigenvector corresponding to the eigenvalue λ1 , which satisfies  x a1  λ1 a1 . By projecting the x onto the n − 1 dimensional subspace perpendicular to the a1 , a similar problem of n − 1 dimension can be obtained, and the projection direction of the largest variance should be a  a2 . If further projections are to be made, and so on, until the projection m times, the same K-L transform definition can be obtained. It can be seen that the K-L transform can reflect the dimensionality reduction information of high-dimensional primitive information, and then projected to the reduced dimension space of the principal component, so that the variance of each component reaches the maximum.

2.6.2.3

Specific Steps to Identify

By using the principle and method of principal component analysis, the recognition process of the moving direction indicator image can be generally included: image preprocessing, import system training sample set, training sample eigenvalue and eigenvector calculation, capture the driving direction indicator and import test samples, the test image feature vector calculation and classification recognition. 1. Driving direction indicator image preprocessing Generally, geometric normalization and gray normalization are necessary before feature extraction and classification of moving direction indicator images. Geometric normalization refers to the positioning result according to the direction of travel, which changes the direction of the moving directions in the image to the same location and the same size. Gray normalization refers to the compensation of the image illumination, etc. the illumination compensation can overcome the influence of illumination changes to a certain extent, and improve the recognition rate. For example, “go straight and turn left”, see Fig. 2.42. 2. Import direction indicator image training samples (1) By its size, old and new, light influence factors of brightness, image acquisition angle and distance, and other considerations, the camera images collected must exist obvious difference, so the driving direction of each type of signs are required to a certain number of training samples, the training can achieve

2.6 Automatic Identification Technique for Traffic Signs

97

Fig. 2.42 Road traffic sign legend

enough algorithm accuracy. Take “straight and left turn” signs as an example (see Fig. 2.42), because of the difference of illumination brightness, the gray images collected are different. Because of the difference in angle of collection, the shape of the sign is different, so 15 training samples are taken. The 15 sampling images of the moving direction sign are processed one by one for two-valued image, and then the Fourier transform is carried out, and the transformed frequency domain image is denoise. As shown in Fig. 2.43, that is, the two-valued image of the legend shown in Fig. 2.42 and its Fourier transform frequency domain image. (2) Import Fourier transform training samples of two-valued steering directions, that is, each 2D direction indicator reads the binarization image data in frequency domain and transformed into one-dimensional vector, the direction of travel is different for the shape of the indicator image, select a certain number of images to form the training set. Assuming that the image size after the Fourier transformation is u × v (u and v, respectively, the width and height of the frequency domain images), the number of image samples for the training direction indication flag is n. Order m  u × v, then the training set is a m × n matrix. To illustrate the convenience, assume that the “Straight running” indicator is classified into first categories, the “Left turn” sign is of the second category, the “Right turn” sign is of the third category, “Straight running and left turn” sign for the fourth category, “Left turn and right turn” sign is the fifth category, the “straight running and right turn” sign is of the sixth category (see Fig. 2.39). Each class takes 15 images, then n  15. The frequency domain i image of class l of the heading indication sign can be represented as one dimensional vector xi(l) T  T  (l) (l) (l) xi(l)  xi1  xij(l) xi2 . . . xim

(2.142)

98

2 Vehicle Driving Safety Technology Based on IVP

Fig. 2.43 Two-valued and its Fourier transform frequency domain diagram

In the formula, xij(l) denotes class l, i sample, and j spectrum element value, l  1, 2, . . . , c is the traffic direction indicator image class, and c is the class number of the driving direction indicator image, here, c  6, i  1, 2, . . . , nl is the traffic direction indicator image sample number, and nl is the direction indicator image sample number of class l, take Fig. 2.42 as an example, nl  n  15, j  1, 2, . . . , m calculates the number of spectral elements for each sample frequency domain image. 3. Frequency domain feature extraction of training samples The transform of the frequency domain image according to formulas (2.124) and (2.125) frequency domain feature extraction method is provided, and the frequency domain feature extraction to the press from low frequency to high frequency sequence into one-dimensional vector, a feature vector of the image frequency domain  T pl  λl1 λl2 λl3 . . .

(2.143)

2.6 Automatic Identification Technique for Traffic Signs

99

Among them, λlk represents the frequency of the Fourier spectrum, k  1, 2, . . . sl , λl1 ≤ λl2 ≤ λl3 ≤ . . . Then each sample feature vector is used to form a feature space matrix P with c×c ⎡

λ11  ⎢  λ ⎢ 12 P  p1 p2 . . . pc  ⎢ ⎢ λ13 ⎣ .. .

λ21 λ22 λ23 .. .

⎤ . . . λc1 . . . λc2 ⎥ ⎥ ⎥ . . . λc3 ⎥ ⎦ . . .. . .

(2.144)

And PT P  I 4. The training samples are transformed into feature space by linear transformation The closer to the low-frequency characteristic, the more realistic the real shape can be represented, that is to say, the characteristic values of the main low frequency include more moving direction indicator images and image characteristic information, therefore, it is possible to select the vector space of the eigenvectors corresponding to the previous eigenvalues, which can maximally represent the main information of the moving direction indicator image. The nl freT  (l) (l) (l) quency domain images xi(l)  xi1 (i  1, 2, . . . , nl ) in frequency xi2 . . . xim domain image library can be projected to the feature space, and the projection vector T  (l) (l) (l)  is obtained. (l) ω . . . ω ω i i1 i2 im Select the vector corresponding to the previous sl main eigenvalues from pl   T λl1 λl2 λl3 . . . T  pl  λl1 λl2 . . . λlsl

(2.145)

And corresponding feature space Pˆ with nl × c ⎡

λ11 λ21 ⎢ λ12 λ22 ⎢ Pˆ  ⎢ ⎢ .. .. ⎣ . . λ1sl λ2sl

⎤ . . . λc1 . . . λc2 ⎥ ⎥ ⎥ . . .. ⎥ . . ⎦ . . . λcsl

(2.146)

So there   (l) (l) (l)  (l) 1 2 . . . nl

(2.147)

Therefore, (l) can be used to represent class l direction indicator images.

100

2 Vehicle Driving Safety Technology Based on IVP

5. Collect scene image (1) In driving, the scene image in front of the vehicle is collected in real time, and the geometric normalization and gray normalization are carried out according to the method provided in step 1, then the image is two-valued after the geometric normalization and gray normalization. (2) Use the training samples to centralize the established marker two-valued map as the template, a fast rough matching method for two-valued images of scene images is presented. The so-called fast coarse matching, that is to set a “relaxation threshold”, as long as the test samples to search for a region, the region and the training samples in any one binarization template matching similarity is less than the relaxation threshold, the regional interception down as a test sample. The relaxation threshold is assumed to be σ . It is assumed that the matching similarity values of the 6 types of two-valued traffic direction indicator template and two-valued scene are D1 (p1 , q), D2 (p2 , q), …, D6 (p6 , q) respectively, among them, p1 , p2 , …, p6 are indicator two-valued templates for 6 direction indicators, q is the area intercepted in the two-valued map of the scene image. In the fast coarse matching process, once the relationship exists Dl (pl , q) ≤ σ

(2.148)

In the formula, indicator type l  1, 2, . . . , 6. 6. Import test samples After you get the test sample, repeat step 2, conduct the import of test samples. 7. Calculate the spectrum feature vector of the test sample image Repeat steps 3 and 4 to compute the spectral eigenvalues and eigenvectors of the test sample. The feature vector of the test sample is projected into the feature subspace represented by formula (2.147). At this point, the frequency characteristic of the current direction indicator image is projected to the feature subspace, which corresponds to a point in the subspace, on the other hand, any point in the subspace corresponds to a moving direction indicator image. 8. Recognition of the moving direction indicator image The test samples projected to the feature subspace are compared with the training samples one by one, determine the category of the sample to be identified. For example, the nearest neighbor distance classification function is used for identification, that is *  *  (2.149) G , (l)  min * − (l) * l

2.6 Automatic Identification Technique for Traffic Signs

101

The A here represents the characteristic subspace of the test sample, through the operation of formula (2.149), can confirm, in the test sample, what kind of direction sign image does each kind of driving direction belong to? The identification procedure is then illustrated by the actual road traffic indication sign recognition process, as shown in Fig. 2.44. Among them, Fig. 2.44a is a realtime collection of road images (test samples), Fig. 2.44b is the two-valued result of the test sample, Fig. 2.44c is the matching template for the second “left turn” signs, Fig. 2.44d uses second classes of templates to find the moving direction indicator image area (see dashed lines in Fig. 2.44d) after rapid rough matching, Fig. 2.44e the result of the two valued rough matching region and its Fourier transformation. Through the frequency characteristics of the frequency domain image obtained by extraction and analysis of principal component projection in feature space, can be accurately identified, the roads signs belong to second types of training samples indicating the direction of travel, is shown in Fig. 2.39, that is the “left turn” sign. Sign recognition in the direction of travel over the road, in addition to the image signal processor from the pinhole camera 5, recognition of other categories with recognition algorithm on the ground running direction sign is exactly the same.

(a)

(b)

(d)

(c)

(e)

Fig. 2.44 Sketch map of frequency characteristic extraction of road traffic sign image

102

2 Vehicle Driving Safety Technology Based on IVP

Fig. 2.45 Road traffic indicator legend

As for the automatic identification of other signs of road traffic (including speed limit signs), principal component analysis (PCA) can be used to implement them. For example, the vehicle lane sign (see Fig. 2.45a), no honking signs and trucks banned mark (see Fig. 2.45b). It is no longer repeated here.

2.7 Intelligent Technology to Prevent the Vehicle from Being Hit by the Rear To maintain a certain distance between the front and back vehicles is a problem that must be noticed and grasped for safe driving, especially the car driving on the freeway. However, due to lack of safe driving consciousness or negligence, drivers often cause rear end collision, especially on expressway. Figure 2.46 shows a rear end accident that is common to ordinary vehicles because of improper distance mastery and runaway speed.

Fig. 2.46 Common rear end accident for vehicles

2.7 Intelligent Technology to Prevent the Vehicle from Being Hit by the Rear

103

In order to solve such problems, ultrasound, millimeter wave radar and laser technology have been applied to vehicle mounted system to solve the problem of vehicle distance detection for the front and rear objects. As far as the existing technology is concerned, the ultrasonic measurement distance is shorter. The millimeter wave radar has a great influence on the distance measurement with the interference of electromagnetic wave. Point laser detection points are less. The imaging laser technology (structured light imaging technology) is too complex and the economic cost is too high. Therefore, the promotion of these technologies is limited. It is proved by practice that the image recognition technology is obviously a feasible and reliable intelligent technology.

2.7.1 Device System Composition To prevent vehicle rear end smart technology, except in the front end of the vehicle configuration binocular pinhole camera, but also in the rear of the vehicle configuration binocular pinhole camera, using the signal processor, the original vehicle speed control mechanism and a voice prompt device, constitutes a new intelligent system technology (see Fig. 2.47). After such an overall configuration of the image sensor, the vehicle has the hardware basic conditions to prevent and control the function of the rear end technology. The rear binocular pinhole camera is set on the inner edge of the left and right rear lights of the vehicle (see Fig. 2.48). The image signal processing method of the vehicle signal processing system to the rear binocular pinhole camera is like the pre-binocular camera.

2.7.2 Identification of Trailing Vehicles and Active Control of Their Spacing The rear end collision of the rear vehicle to the vehicle ahead is a more common accident between the vehicle and other vehicles. In particular, the driving process of vehicles often occurs on the freeway. In order to prevent such accidents, it is Front left camera

Rear left camera Signal processor Rear right camera

Front right camera Vehicle speed control mechanism

Voice prompt device

Fig. 2.47 Composition of intelligent inspection and control device for safe distance

104

2 Vehicle Driving Safety Technology Based on IVP

Fig. 2.48 Schematic diagram of the method of setting the binocular camera

very important and practical for vehicles to identify the distance between their own vehicles and the vehicles ahead. In this way, the distance between the vehicle and the front vehicle can be automatically controlled, and even the self driving vehicle can be avoided or the rear vehicle can be warned: keep the safe distance.

2.7.2.1

Identification of Distance Between Front and Rear Vehicles

In order to avoid to be rear end collision of the vehicle by other vehicle, the vehicle needs to observe both the front and the rear of the road at the same time. When it is necessary to avoid the overspeed of the rear vehicle, it is necessary to pay attention to the road conditions ahead of the vehicle and the distance of the vehicle. Therefore, the intelligent technology to prevent the vehicle from being rear ended should always monitor the status of the front and rear vehicles at the same time. Before the control algorithm is realized, the image template on the back of the vehicle is also needed, in addition to the database of the front image template of the vehicle. The following steps of the distance recognition and control algorithm are as follows: (1) Real-time acquisition of the front and rear scene images of the vehicle. The main field of view is used to intercept the observation window according to the size of the most advantageous analysis and operation (see Fig. 2.49). The selection of the window makes the operation of the signal processor can focus on the vehicle situation in the distance of 30–200 m in front and rear of the vehicle. (2) Fast recognition of the front and rear sight of the vehicle. A template of size w × h (see the black dashed box in Fig. 2.49) is applied to match the main field of view window image of size W × H (see the white dashed

2.7 Intelligent Technology to Prevent the Vehicle from Being Hit by the Rear

105

Fig. 2.49 Main field observation window

box in Fig. 2.49). In each matching process, a combination of rough matching and fine matching is used to carry out the search. At the same time, the introduction of “random walk” method: that is a variable step in leaving the search distance jump starting point at the same time, the position of the search may also be up and down around the turns, staggered from the starting point to walk. Matching similarity adopts spatial domain template matching algorithm, that is, template matching in spatial domain. Usually, template is used as filter operator to slide across the whole image to find out the image area that accords with the benchmark. The resulting image can be expressed as R(s, t, x, y) 

w−1  h−1 

f {s(x + m, y + n), t(m, n)}

(2.150)

m0 n0

Among them, x ∈ [0, W − w], y ∈ [0, H − h], the size of the result image is (W − w + 1) × (H − h + 1) (see the principle of template matching in Sect. 2.3.1.1). R(s, t, x, y) is related to the template t(x, y), the field image s(x, y) is related to the current position (x, y). According to the different forms of f {·} operators, R(s, t, x, y) also exists in different forms. The definition of variance SSD(s, t, x, y) is w−1  h−1   2 s(x + m, y + n) − t(m, n) SSD(s, t, x, y) 

(2.151)

m0 n0

The definition of normalized variance is 2 w−1 h−1  n0 s(x + m, y + n) − t(m, n) (2.152) 2 w−1 h−1 2 s(x + m, y + n) · t(m, n) m0 n0 m0 n0

m0 NSSD(s, t, x, y)   w−1 h−1

106

2 Vehicle Driving Safety Technology Based on IVP

The closer the w × h region content of the template and field image is at (x, y), the smaller the error R(x, y) is. After finding the result image of (W − w + 1) × (H − h + 1), as long as the smallest point position on the whole image is found, it can be used as the position of template matching. (3) Determine the lane in which the vehicle is moving ahead. (a) In order to overcome the image irregularity caused by the uneven road turbulence, the real time captured images are rotated. (b) To calculate the row projection center of the front vehicle through the template matching area (see Fig. 2.50). Two intersection points (il , jl ) and (ir , jr ) on the left and right edges are intersected by a horizontal line to the matching area. As a vertical flat split line of two intersection lines, the intersection point of the vertical and transverse axis is M of the line projection center of the front vehicle, and   il − ir ,0 (2.153) M (i, j)  il − 2 The i and j in the form represent the horizontal axis and the longitudinal axis of the observation window image. (c) The image is two-valued and its “skeletonization” is carried by mathematical morphology. (d) Make a horizontal line through the lower edge of the matching area, and find the intersection point a and b of the driving sign line on both sides of the vehicle. The i axis coordinate value M (i) of the center point M is compared with the i axis coordinates of the intersection point a and b to determine the vehicle lane in front of the vehicle (see Fig. 2.50). The i axis coordinates of a and b are a(i)  ia and b(i)  ib , respectively. If Fig. 2.50 Graphic example for calculating the row projection center of a vehicle

( ir , jr )

( il , jl )

a

b

2.7 Intelligent Technology to Prevent the Vehicle from Being Hit by the Rear

107

il − ir < ia 2

(2.154)

b(i) < M (i) < a(i)

(2.155)

ib < il − That is

It shows that the vehicle ahead is on the same lane as its own vehicle. If il −

il − ir > ia 2

(2.156)

That is M (i) > a(i)

(2.157)

It shows that the vehicle ahead is on the left side of the left driving sign line, that is, on the left lane. If il −

il − ir < ib 2

(2.158)

That is M (i) < b(i)

(2.159)

It shows that the vehicle ahead is on the right side of the right driving sign line, that is, on the right lane. It must be pointed out that if a horizontal line is done by the lower edge of the template matching area, because of driving sign line breakpoints, it was temporarily unable to get the intersection of driving sign line, a and b, then the real time images are collected for the next moment, the process of repeating the implementation step (a) to step (d), it is necessary to get the intersection points a and b of the horizontal line and the driving sign line on both sides, and to perform the related operation, realize the confirmation of the driving lane in the front of the vehicle. (4) Calculate the distance between vehicles. First, we use the epipolar line constraint principle to quickly determine any common feature point on the matched target. Then we calculate the distance between the common feature point and its vehicle directly based on the corresponding points Pl and Pr of the common feature points on the virtual imaging plane.

108

2.7.2.2

2 Vehicle Driving Safety Technology Based on IVP

Control of Distance Between Vehicles

When the vehicle in front is in the same lane as its own vehicle, the speed of the vehicle must be controlled to maintain a safe distance. At this time, the vehicle visual sensing system uses the speed intelligent linkage control device and the control instruction generated by the signal processor are responsible to identify the results, speed down the vehicle through the speed control mechanism and keep the distance from the vehicle ahead. The safe distance S is controlled as follows: S

V2 2|g|

(2.160)

In the formula, V is the current vehicle speed of its own vehicle and g is negative acceleration. In the same way, when the rear vehicles are in the same lane with their own vehicles, the speed of the vehicle must be properly controlled to maintain a safe distance. Of course, the g calculated by the formula (2.160) is a positive acceleration at this time. Or, combined with the integrated operation of the distance between the front and back of the vehicle, that is, Sb < Ss ∧ Sa < Ss

(2.161)

At the time of appearance, a reasonable lane change should be used to avoid the rear vehicle’s possible rear end to its own vehicle (see Fig. 2.51). In the form, Sb and Sa represent the current distance between the vehicle and the front, the vehicle and the rear. Ss is the safe distance that must be measured. The so-called “reasonable lane change operation”, that is, through the rear view to grasp the speed of the rear side vehicle, only when the vehicle’s distance between

Left rear camera

Fig. 2.51 Schematic to prevent being rear ended of a vehicle

. .Right rear camera

2.7 Intelligent Technology to Prevent the Vehicle from Being Hit by the Rear

109

front and rear is within the safe range, and the rear vehicle speed is lower than the speed of the vehicle itself, the side way can enter the adjacent lane. At the same time, the vehicle intelligent system will also send a voice warning to the driver in real time. The operation speed is fast (the whole operation cycle of completing image acquisition, processing, recognition and decision is less than 50 ms), software modification and upgrading is very convenient. The recognition accuracy of the system is more than 99%.

Chapter 3

Intelligent Prosecution of Dangerous Driving

The so-called “dangerous driving” includes: drunk driving, the wrong use of accelerator as the brake. All of these are drivers’ mistakes and even dangerous behaviors. With the help of intelligent detection and control technology of vehicles, such accidents can be effectively prevented.

3.1 Intelligent Technology to Monitor Drunk Driving Although drivers’ driving behavior after drinking is condemned by the public, most countries in the world take strict legal measures to stop and punish the behavior of driving after drinking, but because of the existence of personal bad habits, people want all drivers to be able to comply with the traffic conscientiously. It will take a long time. In view of such social conditions, the use of advanced intelligent technology to improve the performance of the car is obviously a very necessary thing, and it is also a very effective technical measure. The intelligent technology that monitors drivers’ driving after drinking in this chapter can accurately monitor and effectively prevent such illegal activities, so it can effectively prevent the occurrence of various forms of traffic tragedies resulting from drunk driving (or drunk driving). The structure of the intelligent monitoring system for driving after drinking is shown in Fig. 3.1. Among them, the most significant technology is alcohol sensing and signal processing. In order to ensure the effectiveness of the monitoring technology, it is necessary to carry out real-time and accurate testing of the alcohol content of the driver.

© Shanghai Jiao Tong University Press, Shanghai and Springer Nature Singapore Pte Ltd. 2019 X. Zhang and M. M. Khan, Principles of Intelligent Automobiles, https://doi.org/10.1007/978-981-13-2484-0_3

111

112

3 Intelligent Prosecution of Dangerous Driving

DC power supply

Conditioning amplifier

Signal processor

Controller

Servomechanism

Voice prompt

Alcohol sensor Auxiliary sensor

Fig. 3.1 Block diagram of drunk driving control system

3.1.1 Alcohol Sensing Theory Alcohol sensor is one of the gas sensors. The gas sensor is an important branch of the sensor field. It is a device that identifies the gas species and transforms it into electrical signals. It is an ideal probe for gas quantitative or semi-quantitative detection, leakage alarm, control, etc. The semiconductor gas sensors are classified from the constituent materials, including metal oxide semiconductor gas sensors and organic semiconductor gas sensors. Metal oxide semiconductor sensor works according to its signal output mode, and can be divided into two types of resistance and non-resistance type. An alcohol sensor is a metal oxide semiconductor sensor. Semiconducting materials sensitive to alcohol (C2 H5 OH) are such as: Pd-La2 O3 -SnO2 and Pd-La2 O3 -WO, etc. The alcohol sensor and its device represent the sensing device for alcohol gas, referred to as alcohol sensing (The following description is the same as this).

3.1.1.1

Gas Sensing Mechanism of Alcohol Sensor

In the process of semiconductor gas sensor, the interface potential changes firstly go through the adsorption of the surface on the base material to the surrounding gas, then, through the oxidation and reduction reaction, from which the conductivity change of the gas sensitive to the material is formed, afterwards the chemical signal is changed into an electric signal to achieve the purpose of detection. In other words, when the surface of an oxide semiconductor adsorbed a certain gas, the surface energy level formed by the adsorbed gas on the surface of the semiconductor is not at the same level as the Fermi level of the semiconductor itself. Therefore, a space charge layer is formed near the surface of the semiconductor. The so-called Fermi energy represents the highest energy levels of the electrons in the ground state system of metals at temperatures 0 K. The “Fermi energy level of the semiconductor itself” refers to a type of oxide semiconductor exposed to the atmosphere, such as Pd-La2 O3 -SnO2 , etc. The surface is always adsorbed with a certain amount of electron donors (such as hydrogen atoms) or electronic acceptors (such as oxygen, hydrogen atoms), thus forming the size of the space surface charge. Because of the difference in surface energy levels,

3.1 Intelligent Technology to Monitor Drunk Driving

113

a barrier is formed, thereby forming a space charge layer near the surface of the semiconductor. The position of the surface energy level relative to the Fermi energy level of the semiconductor depends on the electrophilic nature of the adsorbed gas. If its electrophilic properties are low (i.e., reduced gases), the resulting surface energy levels will be below the Fermi energy level, and the adsorbed molecules provide electrons to the space charge region and become positive ions adsorbed on the semiconductor surface. At the same time, the conductivity of the charge layer increases correspondingly due to the increase of the electron concentration in the space charge layer. On the contrary, if the electrophilicity of adsorbed gases (i.e. high oxidizing gas), generated surface energy is located above the Fermi energy level of adsorbed molecules absorb electrons from the space charge region and become negative ion adsorbed on the semiconductor surface. At the same time, the conductivity of the charge layer decreases correspondingly due to the decrease of the electron concentration in the space charge layer. Therefore, when the concentration of gas changes on the surface of the semiconductor, the conductivity of the space charge region changes. That is to say, the conductivity of the space charge region is modulated by the type and concentration of gas. Specifically, the conductive carriers of semiconductors such as Pd-La2 O3 -SnO2 are electrons, and the surface conductivity changes g can be given by the following formula g  eμn

(3.1)

In the formula, e represents the electron energy, μ represents the surface electron mobility, and n represents the surface carrier density change. In the space charge layer with a thickness h, n can be obtained by integration h n 

(n(z) − nb )dz

(3.2)

0

In the formula, nb is the carrier concentration inside the semiconductor, and n(z) is the carrier concentration in the space charge layer. For the space charge layer of a rectangular parallelepiped, the surface conductance changes to G 

gA L

(3.3)

In the formula, G is the surface conductance change of the space charge layer, A is the cross-section area of the semiconductor component, and L is the length of the semiconductor component. Similarly, it is assumed that the sensitive material constituting the component is a continuous thick film, for the regular rectangular oxide semiconductor device, the internal conductance Gb is

114

3 Intelligent Prosecution of Dangerous Driving

Gb  enb μb A

H L

(3.4)

In the formula, μb is the internal electron mobility of semiconductor, and H is the total semiconductor thickness. If the electron mobility is very less affected by the state of the oxide semiconductor surface, i.e., μb ≈ μ, the change in the relative conductance of the oxide semiconductor can be derived by formulas (3.1), (3.3) and (3.4) G n  Gb nb H

(3.5)

The relative conductance change G of an oxide semiconductor gas sensor is the Gb detection sensitivity of the element. Therefore, from formula (3.5) it can be seen that in order to improve the sensitivity of gas sensors, oxide materials with low carrier concentration nb must be selected, and the thickness H of gas sensors should be reduced as much as possible. For gas sensors that have been identified as materials and their structures, its sensitivity depends on the change n of carrier concentration on the surface of oxide semiconductors that can be introduced by a unit concentration of gas.

3.1.1.2

Alcohol Sensor Structure

The structure of the alcohol sensor consists of Pd-La2 O3 -SnO2 semiconductor (referred to as two tin oxide semiconductor), heating wire, metal shell, stainless steel net cover, base, heating electrode and sensing output port. Among them, the core components are sensitive components, two tin oxide semiconductor and heater for activating the sensitive element. As shown in Fig. 3.2, a diagram of the structure of the alcohol sensor is shown in figure (a), figure (b) is a physical picture of the alcohol sensor TGS822. The two-stannic oxide semiconductor sensitive element in the alcohol sensor must be activated only when it is heated. At this time, as long as there is alcohol vapor spreading around the two-tin semiconductor, it produces a fuel effect, and a high potential output is formed on the sensing output port of the two tin oxide semiconductor.

(a)

(b) Stainless steel mesh cover

Two tin oxide semiconductor

Metal case Heating wire

Base Sensor output head Heating electrode

Fig. 3.2 Schematic diagram of alcohol sensor

3.1 Intelligent Technology to Monitor Drunk Driving

3.1.1.3

115

Signal Generation Principle of Alcohol Sensing

As far as the commonly used two stannic oxide semiconductor alcohol sensors are concerned, it belongs to N + N composite structure semiconductor gas sensor. The physicochemical properties of two stannic oxide semiconductor alcohol sensors are determined by their material and structural characteristics. 1. Relationship between material resistivity and gas concentration For N-type semiconductors, when the surface is accumulated by electrons, the surface conductivity of the grains can be described by the following formula. g  eμn

(3.6)

The role of the hole is neglected here. Considering the presence of a large number of grain boundaries in powder materials, these factors are taken into account, and the total conductivity gΣ of the material is   eVs (3.7) gΣ  α exp − kT In this formula, α is an influencing factor related to the contact area of the grain and the diameter of the grain, and the formula (3.7) is changed into the resistivity representation, here is   eVs (3.8) ρ  ρ0 exp kT In this formula, ρ  g1Σ , ρ0  α1 , Vs is the barrier height for the semiconductor surface adsorption equilibrium, k is Boltzmann constant, and T is the thermodynamic temperature. Assuming, Qs is the amount of charge adsorbed on the unit area of the semiconductor surface, given by      eV (x) kT 2 −1 (3.9) Qs  2eε (ND − NA )Vs + (nb + Pb ) exp e kT In the formula, NA and ND are ionization acceptor and donor concentration respectively, Pb is semiconductor internal pressure, V (x) is surface layer barrier height, and ε is correction factor. After the formula (3.9) is deformed, it can be obtained     Qs2 eV (x) nb + Pb eVs exp + −1 (3.10)  2εkT (ND − NA ) kT ND − NA kT

116

3 Intelligent Prosecution of Dangerous Driving

This is the equation between the surface charge of semiconductor and the surface barrier. Considering that when x → 0, V (x)  0, so the formula (3.10) at this point becomes Qs2 eVs  kT 2εkT (ND − NA )

(3.11)

The formula (3.11) is substituted into the formula (3.8), here is   Qs2 ρ  ρ0 exp 2εkT (ND − NA )

(3.12)

Namely ln ρ  ln ρ0 +

Qs2 2εkT (ND − NA )

(3.13)

Qs is the total charge generated by the adsorbate in various positions, that is, the charge of the adsorbate: Qs 

M

Ns,i qi Qi

(3.14)

i1

In the formula, Ns,i is the density of the semiconductor surface at the i position, qi is the unit charge of the adsorbed ion, Qi is the surface coverage, and it is a function of the partial pressure (concentration) of the gas. Therefore, the formula (3.13) can be written again M

ln ρ  ln ρ0 +

2 Ns,i qi Qi

i1

2εkT (ND − NA )

(3.15)

Formula (3.15) is the relationship between the resistivity of alcohol sensitive material and the concentration of gas. 2. Alcohol sensor equivalent circuit The equivalent circuit of the alcohol sensor is shown in Fig. 3.3. As shown in the figure, the gas sensors composed of two N type semiconductors are equivalent to two equivalent resistances, given as Ra and Rb . Due to the difference between the doping material and the proportion in the development process, the semiconductor A corresponding to Ra is particularly sensitive to a specific gas (alcohol), semiconductor B corresponding to Rb is insensitive to specific gases (alcohol), thus, in a heating environment, when the alcohol containing gas is attached to the sensor semiconductor material, the resistivity of the semiconductor A varies according to formula (3.15)

3.1 Intelligent Technology to Monitor Drunk Driving

Heating voltage

Vh

117

Ra

Working voltage

Rb

Sensor output voltage

VC

V

Fig. 3.3 Alcohol sensor equivalent circuit

as the concentration of the gas changes. Relatively speaking, the resistivity of semiconductor B is less affected by the concentration of gas, showing almost unchanged state. As can be seen from Fig. 3.3, the output voltage V of the sensor in a clean gas environment is expressed as V0 

Rb VC Ra + Rb

(3.16)

In the formula, VC is the working voltage of the gas sensor. As long as the alcohol gas is present, the alcohol sensor will yield an output voltage and can be expressed as V 

Ra

Rb VC + Rb

(3.17)

The sensitivity βa of the semiconductor A is defined as βa 

Ra Ra

(3.18)

The sensitivity βb of semiconductor B is βb 

Rb Rb

(3.19)

The sensitivity β of the alcohol sensor is β

V V0

(3.20)

After the formula (3.16) and formula (3.17) are introduced into formula (3.20), we can get the following equation β

Rb Ra βb + RRab βa

1+

(3.21)

118

3 Intelligent Prosecution of Dangerous Driving

Fig. 3.4 Equivalent circuit of modern gas sensor Heating voltage

Vh Sensor output voltage

Thus, when the sensor contacts the gas to be tested, if βa > βb , βb , ββab

βb βa

V

< 1, then

> 1, then β < 1. A  1. When the sensor contacts the interfering gas, if βa < Generally, the existing alcohol sensor have an initial relaxation time, i.e., when the alcohol sensor is connected with a heating voltage, the semiconductor resistance will change from big to small, after a certain period of time it rises gradually and tends to a stable value. The time required to stabilize from the ohmic heating to the resistance is called the initial relaxation time, which is also referred to as an initial stabilization time. The initial relaxation time brings a lot of inconvenience to the application. For N + N type gas sensor, when the composition of the two kinds of sensitive A and B gases, the sensitive element of the whole early relaxation characteristics are nearly equal, the output signal voltage of whole gas sensor is composed by these which can be quickly stabilized and also can effectively shorten the components of the initial relaxation time to improve the initial stability of components. It must be pointed out that, after years of research and improvement, the current alcohol sensor in the structure can be very accurate, small, and, in the actual signal processing circuit of semiconductor gas sensor has no additional working voltage VC , which has been able to activate in the heating state, to produce high potential output due to the sensitivity of attached gas, is as shown in Fig. 3.2, the equivalent circuit as shown in Fig. 3.4.

3.1.2 Signal Processing Method for Alcohol Sensor Considering the need for a gas sensor when working to heat the gas sensitive semiconductor, the life of a gas sensitive semiconductor will shorten obviously under heating condition, and alcohol sensor is not an exception. Therefore, under the premise of ensuring that the alcohol sensor can work normally, we should also take some technical measures to shorten the heating time of sensitive materials to minimum level.

3.1.2.1

Alcohol Sensing Signal Processing System

Alcohol sensing signal processing system, intelligent monitoring system structure diagram that is also known as sensing system of drunk driving, as shown in Fig. 3.5, including passive infrared sensor, alcohol sensor, conditioning amplifier 1, condition-

3.1 Intelligent Technology to Monitor Drunk Driving Passive infrared sensor D Human radiation infrared

S G

Power adapter 1

Electronic starter switch

Conditioning amplifier 1 Signal processor Power adapter 2

Alcohol sensor

119

Controller

Voice prompt

Alcoholic gas Conditioning amplifier 2

Vehicle speed control mechanism

Fig. 3.5 Schematic diagram of alcohol sensing signal processing system

ing amplifier 2, signal processor, controller, electronic starting switch, voice prompt device, speed control mechanism, power adapter 1, and power adapter 2. Among them, passive infrared sensors are used to sense the infrared radiation of the human body. That is to say, the passive infrared sensors have the ability to passively receive radiation from the human body (or other objects that radiate infrared radiation) and converts it into electrical signal output. Passive infrared sensors have three electrodes, D, S and G. The D is connected to the positive power supply, the G is grounded, and the S is the electrical signal output pole. When the infrared sensor is not equipped with any guided wave device, the passive infrared sensor has wide infrared wave to receive the three-dimensional angle. The conditioning amplifier 1 includes a two-stage voltage amplifier. When the input of the conditioning amplifier receives the infrared electrical signal output from the infrared sensor, the original signal is amplified through the first stage voltage amplifier, the amplified signal is transmitted to the second stage by a resistance capacitance coupler and the voltage amplifier continues to be amplified, the signal input after the second stage amplification is input to the signal processor input port. The circuit structure of the conditioning amplifier 2 is the same as the conditioning amplifier 1, but the static working points of the two amplifiers are different, and the conditioning amplifier 2 is used for amplifying the electrical signals output from the alcohol sensors. The signal processor includes two analog signal input channels, two analogs to the digital conversion module, operation module and output interface. Two analog signal input channels are respectively connected with the two output ports of the amplifier, with infrared and gas (alcohol) sensor output signal respectively, two analog-todigital conversion modules were converted to the corresponding digital signal. The digital signals represent two sensing information, after the operation and the decision of the operation module, form corresponding control instructions and output to the controller and digital speech memory in parallel through the output interface.

120

3 Intelligent Prosecution of Dangerous Driving

The voice prompt comprises a digital instruction input interface, a switch signal input interface, a decoder, a digital speech module, a digital to analog converter, a power amplification module and a loudspeaker. Among them, the digital voice module is composed of speech units, each speech unit is used to store digital speech coding. The signal processor through the internal data bus call voice unit to realize the synthesis of digital speech encoding, which consists of a digital voice signal through an input port with complete semantic voice reminders are sent to the voice prompting device. When the voice prompts digital command input interface, it receives the control instruction signal output from the processor. After the decoder interpretation, speech unit linked digital voice module. In order to link transmission DAC, the speech unit sequence converts voice analog signal, then the voice analog signal input to the power amplifier module with power amplifier after driving the speaker sends out the corresponding voice warning. The power adapter 1 is set up by passive infrared sensors, two conditioning amplifiers, signal processors, controllers, digital voice storage, voice players, and speed control mechanisms, it can provide the corresponding working voltage according to the different circuit work requirements. The power adapter 1 is controlled by an electronic start switch, and the electronic starter switch is controlled by the controller. The power adapter 2 is specially set up for the heating circuit of the alcohol sensor, and its switch mode is directly controlled by the signal processor. The electronic switch device includes the first-gear machine/electrical switch module, the second-gear alcohol sensing control switch module and the third-gear engine starting switch module. The grade switch of the electronic starting switch is realized by the mechanical rotation of the starting key of an automobile. The electronic starting switch of each grade function includes clockwise first-gear, the mechanical switch to open the first power adapter, passive infrared sensor, two conditioning amplifiers, signal processor, a controller, a voice prompt device, electronic devices such as a vehicle start switch detection and control system for equipment supply. It continues to rotate second times in clockwise direction, and the main channel of the alcohol sensing control switch module is turned on. It continues to rotate third degrees clockwise, that is, at the ready to start the engine working state. The first-gear mechanical/electrical switch module and the third-grade engine start electric switch module are all original automobile electric switch module. The second-grade alcohol sensing control switch module is composed of a bidirectional thyristor. The source and drain poles of the bidirectional thyristor are connected in series with the third stage engine starting circuit, the control part of the bidirectional thyristor relates to the output interface of the controller. When the signal processor determines a certain concentration of alcohol vapor filled from the cab, the controller output low voltage, the bidirectional thyristor is cut-off (conduction), the prohibition of third grades of automobile engine start. Otherwise, the controller outputs a high voltage to ensure that the bidirectional thyristor is in the conduction state and can perform the normal startup of the third-grade automobile engine. When the electronic switch starts the first level switch, the onboard check system starts to work, and the first power adapter supplies power to the corresponding device. Afterward when the electronic starter switch switches on the second level switch, it

3.1 Intelligent Technology to Monitor Drunk Driving

121

will be controlled by the alcohol sensing signal. As long as the driver is drunk, the output instructions of the device system will prohibit the electronic starting switch. The switch on the third grade cannot cause the starting motor and its auxiliary circuit to have a working power supply. The speed control mechanism comprises of an input interface, a first digital to analog converter, a power amplifier, an electromagnetic valve and an electric brake push rod. When the vehicle is in the normal operation, the solenoid valve of the speed control mechanism is at the full open state, that is, the opening is 100%. Once through the detection of an alcohol concentration in the cab exceeds the approved threshold, even if the vehicle has been started, the signal processor in a control instruction under the action of electromagnetic valve will produce the corresponding real-time magnetic pull valve to reduce the original opening, thus reducing the flow of fuel and forcing the vehicle deceleration. At the same time, the electric brake push rod coil is under the influence of an input voltage signal, and the axial thrust force is generated on the push rod, and the foot brake is driven by the torque transmission of the rod and bar mechanism, so that the vehicle gradually slows down and finally stops. If the driver uses a non-drinker to blow gas or replace the driver during the midcourse, the above system will not make such a fraud happen. Because the alcohol sensor is always in working condition monitoring, starting the car, the device to detect the system once the cab of the alcohol is still beyond the approved threshold, through the integration of auxiliary and infrared sensing information is calculated and confirmed, real time signal processor to voice prompting device and a speed control mechanism to send control command. Under the influence of the control command, the driver is warned not to drive, and the vehicle is automatically slowed down to the brake.

3.1.2.2

Method for Setting Sensor

1. Infrared sensor settings Under normal circumstances, the passive infrared sensor can be mounted on the driver’s cab, facing the driver’s head to receive infrared radiation from the head of the human body, as shown in Fig. 3.6. The size of the infrared receiving angle can be changed by adjusting the focusing convex film (the resin convex lens) at the front end of the infrared sensor. Thatswhy, as long as changing resin focal plate with different focal length, we can change the infrared range that is the infrared sensor can sense from the surrounding body to its radiation. According to the height of the top of the cab, it is advisable to control the infrared receiving angle. 2. Alcohol sensor setting method In general, the alcohol sensor is located near the central region of the steering wheel. As shown in Fig. 3.7, the advantage lies in

122

3 Intelligent Prosecution of Dangerous Driving

Fig. 3.6 Infrared sensor settings method

Fig. 3.7 Alcohol sensor setting method

Passive infrared sensor

Exhaled gas

Receiving angle

Alcohol sensor

(1) It is closest to the driver’s face, and it is easy to sense the air exhaled by the driver. (2) It is easy to install and maintain. (3) It is easy to link with the signal processor and the electronic starter switch. The induction threshold of the alcohol sensor is the gas alcohol concentration that can be determined by the amplification gain of the conditioning amplifier and the mathematical model of the signal processor operation.

3.1.2.3

System Control Principle

When the electronic start switch is at the first switch “open” position, the intelligent control drinking driving system enters into the working state. The working process of the system is as follows.

3.1 Intelligent Technology to Monitor Drunk Driving

123

(1) Initialization state, namely power adapter 1 is turned on, infrared sensor, amplifier (including amplifier 1, 2), signal processor, controller, speed control mechanism and a voice player in working state. (2) The signal processor confirms that the person is seated in the driver’s seat according to the infrared radiation of the human body received by the infrared sensor. (3) The signal processor sends the first control instruction to the power adapter 2. The control instruction manages the electronic switch in the power adapter 2 to open a large current fast heating channel. Therefore, the heating wire of the alcohol sensor is used to quickly heat the alcohol (gas) sensitive element under the action of large current. (4) After a certain time delay t1 (such as t1  0.5 ∼ 2s), the signal processor sends second control instructions to the power adapter 2. The instruction is to turn off the electronic switch on the large current channel and open the electronic switch on the small current (rated maintenance current) channel. Therefore, the heating wire of alcohol sensor continues to heat the alcohol (gas) sensitive element under low current, and maintains the stability of working temperature, which is called the normal heating alcohol sensor. (5) The alcohol sensor begins to identify the gas that is attached to the sensing element. That is, if the gas concentration of the adsorbed gas exceeds the sensitivity, a high potential output is produced at the output of its electrical signal. The voltage signal formed by the high potential to the ground is fed into the first analog signal input channel of the signal processor after the conditioning amplifier is fed into the signal processor. (6) After calculating the received alcohol sensing signal, the signal processor determines whether the gaseous alcohol content from the detected person is “over standard” according to the decision threshold. If so, the third control instructions are generated, which are output in parallel to the controller and the digital speech memory, and the step (7) is executed. Otherwise, the signal processor delays a certain time of t2 (such as t2  2s), and the system then performs the following steps (9). (7) According to the control instruction, the second switch controller makes the electronic starting switch close, it can not enter the engine start preparatory work, effectively prohibits the attempt to drunk driving. At the same time, digital voice storage for voice synthesis, and through the voice player to inform the driver, such as: “please do not drink during driving!” (8) The signal processor determines whether the drinker is out of the cockpit by infrared radiation of the human body received by the infrared sensor? If it does not leave, the processor of the signal processor continues to execute the step (7) until the “drunk driver” leaves the cab. Otherwise, the signal processor interrupts third control command output, namely the electronic starting switch locking the second switch is released, control the voice prompting device to stop playing “warning” system back to the initialization state.

124

3 Intelligent Prosecution of Dangerous Driving Close the fuel door Yes

Detected person left?

Start

No

Yes Excessive alcohol?

No

Yes

No

Engine normal start

Is there anyone in driver's seat?

Alcohol sensor heating closed

Maintain normal running

Quick heating alcohol sensor Detected person left? Engine start switch unlock

Delayed t1

No

Yes

Yes No

Detected person left? Normal heating alcohol sensor

Alcohol sensor heating closed

No Excessive alcohol?

Speech warning

Yes Engine start switch latch up

Fig. 3.8 Flow chart of intelligent control drink driving

(9) Signal processor interrupt third control command output, namely the electronic starting switch locking the second switch is released, to control the voice prompting device does not play “warning” to allow electronic switch to start to enter the third grade of automobile engine start work. The system continues to test the gases emitted by motorists. If at this time, the concentration of alcohol in the cab there continue to exceed the standard phenomenon, namely the alcohol concentration calibration beyond the threshold, the signal processor to the speed control mechanism fourth sends control command, the control instruction under the action of the vehicle began to close the throttle, automatic deceleration until the brake, and the central monitoring system of the present invention to initialize the working state. Therefore, it can effectively prevent the occurrence of illegal activities, such as drinking, drivers borrow normal people to start their vehicles, and in the way of replacement, so that drinkers (or drunk) re driving vehicles. The whole procedure described above can be described in the system program flow shown in Fig. 3.8.

3.2 Prevent Mistake of Stepping on Accelerator When Stopping Brakes

125

3.2 Prevent Mistake of Stepping on Accelerator When Stopping Brakes Suddenly some special situations appear in the front of a moving vehicle (such as someone suddenly crosses the road, or the vehicle suddenly brakes on the front and so on), due to the stress of the driver, he should have made an emergency brake, but he wrongly trampled the accelerator, resulting in the accident of vehicle hitting pedestrians or vehicle rear end. Such events are very common. Due to the mistake of trampling on the throttle while driving on the brakes, the heavy loss of life and property is one of the most serious and frequent traffic accidents at present. This is not an exception at home and abroad. Although the automotive visual technology has been introduced earlier, it can automatically identify pedestrians or vehicles that appear in front of a vehicle, and perform an intelligent technology for effective avoidance. However, it can not exclude the mistake of treading on the throttle of the driver when the driver’s brakes are caused by emotional tension and so on. Therefore, as a vehicle with perfect performance, it should have the intelligent technology of “urgent brakes to prevent the mistake of stepping on the throttle”.

3.2.1 Anti Misoperation Device for Emergency Brake Anti misoperation device for the emergency brake, the device comprises a signal processor, a controller, a flow measuring and controlling electromagnetic valve, a first electronic switch, a second electronic switch, and an electric push rod and a brake, as shown in Fig. 3.9. The input port of the signal processor is connected to the differential pressure electric signal output port of the flow measuring and controlling electromagnetic valve. The output port of the signal processor is connected to the input port of the controller. The first output port of the controller is connected to the first electronic switch control pole. The second output port of the controller is connected to the second electronic switch control pole. The source/drain poles of the first electronic switch are connected in series with the DC power supply circuit of the flow measuring and controlling electromagnetic valve. The flow measuring and controlling electromagnetic valve is arranged on the upper side of the pedal gas in the oil supply pipeline of the automobile. It allows the car to supply oil, first through the flow can be measured after the solenoid valve, and then pedal the throttle to the engine cylinder. Second, the electronic open/off source/drain poles are connected in series with the DC power supply circuit of the electric push rod. The upper end of the push rod of the electric push rod and the lower arm of the foot brake rod form a mechanical connection which can rotate in a plane angle range of 90°.

126

3 Intelligent Prosecution of Dangerous Driving

Second electronic switch Signal processor

Controller

Electrically driven push rod

First electronic switch

Brake

24V Controllable electromagnetic valve

Pedal fuel valve

Fuel

Fig. 3.9 Schematic of device that prevents the accelerator from being used as a brake

Fig. 3.10 Flow measuring and controlling electromagnetic valve. 1 Solenoid valve body, 2 Valves, 3 Compression springs, 4 Electromagnets, 5 Upstream pressure guide holes, 6 Downstream orifice tubes, 7 Differential pressure sensors, 8 Plastic soft pressure plate, 9 Cable interfaces, 10 Electromagnet coil, 11 Input wire, 12 Output wire, 13 Upstream pipe joint, 14 Downstream pipe joint, 15 Base nut cap, 16 Leakproof gasket

3.2.1.1

Device Structure Features

The braking device introduced here is used to prevent mistakenly stepping on the fuel oil device. The most prominent feature is the adoption of an invention patent technology, that is, the flow can be measured and controlled by solenoid valve. The flow measuring and controlling solenoid valve comprises an electromagnetic valve body, a valve, a compression spring, an electromagnet, a pressure guide hole tube, a differential pressure sensor, a plastic soft pressboard and a cable interface, as shown in Fig. 3.10. Among them, valves, compression springs, electromagnets, differential pressure sensors, plastic soft pressboard and cable interfaces are distributed inside the solenoid valve body and arranged in order from the bottom to the top. The valve is in the center

3.2 Prevent Mistake of Stepping on Accelerator When Stopping Brakes

127

of the solenoid valve body. The compression spring is sheathed on the cylinder on the upper part of the valve to produce a prestress to the valve so that the solenoid valve is in the normally open state. The electromagnet is located at the top of the cylinder and has a gap with the cylinder. The pressure input port of the guide hole-tube is located on the pipe wall of the fluid channel inside the solenoid valve body. The pressure output port of the guide hole-tube relates to the pressure input port of the differential pressure sensor. The differential pressure electrical signal of the differential pressure sensor is output through the output wire. The input wire of the electromagnet coil and the differential pressure sensor’s fluid differential pressure electrical signal output wire relates to the cable interface. They are connected to the controller and the signal processor outside the solenoid valve, respectively. The electromagnetic valve body is a shell structure. That’s why, it is hollow from inside, and there are an upstream pipe joint and a downstream pipe joint on both sides of the outer wall. The solenoid valve body depends on the outer thread of the upstream and downstream pipe connections to ensure a reliable connection with the upstream and downstream pipelines. The upstream pipe joint and the downstream pipe joint form a fluid passage with the valve passage. The bottom of the solenoid valve body is equipped with a bottom screw cap to facilitate the loading and unloading of the valve and the compression spring. The solenoid valve is divided into two parts, the upper part and the lower part, and the upper part is placed with an electromagnet, differential pressure sensor, plastic soft platen and cable interface from bottom to top. The electric signal output line of the lead wire of the electromagnet coil and the differential pressure sensor are welded by the wire hole on the plastic soft platen and the pin in the cable interface, respectively. The cable interface relates to the solenoid valve body by screw thread. The compression spring, the valve and the bottom nut cap are placed in the lower part of the solenoid valve body from top to bottom. The internal geometry of the solenoid valve ensures that the valve can move smoothly in the radial direction to change the flow area of the valve from the maximum to the minimum, or from the minimum to the maximum. The valve is a moving part of the solenoid valve body. The compression spring is a half moving part and the others are stationary parts. The valve is a step type cylinder structure, the upper part is a cylinder, the lower part is a cuboid, and a flow passage is arranged on the direction of the cuboid to the liquid flow. The flow path can be conical. Its exit radius is smaller than the inlet radius, and the inlet radius is the same as the upstream pipe joint radius of the solenoid valve body. With the help of the gradual contraction of the conical flow passage, the resistance to fluid flow is strengthened. Therefore, when the fluid flows through the conical passage, the flow differential pressure will appear in the upstream and downstream of the channel. When the valve is in the open state, the flow area is fully opened to the fluid. As long as the upper cylinder of the valve is moved up and down, the flow area of the valve can be changed until the valve is closed. With the step cylinder structure of valve, the compression spring is set on the upper cylinder of valve, so that the cylinder is normally opened with the prestress of the compressed spring. Once the electromagnet is charged, the electromagnetic force produced by the electromagnet pulls the upper cylinder of the valve to overcome the thrust of

128

3 Intelligent Prosecution of Dangerous Driving

the compression spring, so that the valve moves toward the closed direction until it closes. When the electromagnet is energized then it loses power, the electromagnetic force of the electromagnet will disappear. At this point, the valve will move in the direction of opening until the full opening is under the action of the thrust of the compression spring. The electromagnet consists of two parts, the wire coil and the soft magnetic core, which are installed on the upper part of the valve, and it keeps a certain static distance from the upper cylinder of the valve. The wire coil is sheathed on the soft magnetic core. When the wire coil is charged, under the action of the electromagnetic field, the soft magnetic core generates magnetic induction and forms a magnetic force to overcome the thrust of the spring to pull the magnet of the valve, and then gradually close the valve until it is fully closed. On the contrary, the upper cylinder of the valve moves under the thrust of the compression spring until the valve is opened completely. The wire coil is powered by 24 V DC power supply. When the series hard switch controls its turn-on or turn off, the wire coil has only two working states, i.e., the maximum current is received, or the power is not switched off, which is called the hard switch working state. Through the cascade control the electronic switch is turned on or off, the soft opening and soft switching can use electronic switching devices off characteristics, realize continuous change of the receiving coil current, can form a smooth flow of continuous adjustment, but also can avoid the water hammer effect flow area of solenoid valve when there is mutation. The pressure guide-hole tube is two diameter 1 mm small hole tube, plays the role of conduction fluid pressure, one of them is the upstream orifice pipe, and the pressure input port is located upstream of the fluid channel near the valve inlet wall, the other is the downstream orifice pipe, and the pressure input port is located downstream of the fluid channel near the valve outlet. The pressure output port of the upstream and downstream pressure-guide-holes is connected to the pressure input port of the differential pressure sensor respectively. Differential pressure sensor differential pressure electrical signal through the wire output. When there is fluid passing through the valve, the throttling effect is caused by the shrinking shape of the flow passage of the valve, resulting in different fluid pressures at the inlet and outlet of the valve. The fluid pressure is transmitted from the conduction pressure hole-tube to the input port of the differential pressure sensor, then the differential pressure sensor is transformed into differential pressure electrical signal by the differential pressure sensor, and finally output by the output wire for a subsequent module or device processing. The plastic soft pressboard can make use of its insulation characteristic to ensure the good insulation between the wire coil lead wire and the differential pressure sensor electric signal output line, it also uses its soft plasticity, with the cable interface screw connection of the fastening force to ensure the internal components of the solenoid valve fixed. The cable interface is connected tightly with the upper part of the electromagnetic valve body by connecting threads, and the signal lines are numbered in order to facilitate the signal exchange with the other technical equipment outside the communication cable through the communication cable.

3.2 Prevent Mistake of Stepping on Accelerator When Stopping Brakes

129

Hydraulic brake device

Electric putter Fig. 3.11 Working principle diagram of electric putter

The bottom nut cap ensures the convenience of loading and unloading of the valve and the compression spring, as well as the cleaning of the valve interior. The bottom nut cap is provided with a leakproof washer to prevent the liquid from leakage. In addition, there is an important part of the device that is an electric putter. This electric putter, the maximum thrust can reach 300 N, the maximum stroke can reach 130 mm. The electric putter is driven by a 24 V/12 V linear permanent magnet motor. The full load current is 2 A at 24 V and the response speed is 27–50 mm/s. See Fig. 3.11, the electric putter is installed on the lower pedal arm after the brake plate extension. Once the signal processor determines that the driver mistakenly treads the fuel pedal, the signal processor immediately sends brake instructions to the controller. At the same time, when the controller cuts off the fuel channel, the controller outputs a trigger signal through the second output port to the second electronic switch to conduct the second electronic switch, and then starts the electric putter. Driven by a linear motor, the thrust generated by the push rod is transmitted by torque, resulting in the automatic braking of the car. Therefore, the possible accidents can be avoided. The device has been reformed to the original foot brake mechanism, the original gravity arm is used as the thrust arm of the electric push rod, which does not affect the original foot brake function.

3.2.1.2

Device Operation Principle

When the vehicle is in the running state, the signal processor receives the differential pressure signal of the oil flow which is transmitted by the flow measuring and controlling electromagnetic valve in real time, the transient flow value of the oil circuit is calculated according to the differential pressure electric signal of the fluid, and the current flow is compared with the flow at the previous moment in real time. Once the instantaneous change rate of flow exceeds the preset threshold, it shows that the

130

3 Intelligent Prosecution of Dangerous Driving Flow signal Signal processing

Flow controllable solenoid valve

Shut-off valve Brake

Yes Flow mutation?

Control output

Electric putter

No Continue to detect flow

Fig. 3.12 Working principle of anti misoperation device for emergency brake

driver has an abnormal condition, because he hurriedly and fiercely trampled the accelerator and thought he was trampling on the brakes. That’s why, the driver had to tread on the throttle when he had to brakes. As a result, the signal processor immediately sends a brake instruction to the controller. The controller outputs the trigger pulse to the control pole of the first electronic switch immediately through the first output port according to the instruction, and the first electronic switch is opened. At the same time, the flow-controlled solenoid valve gets current through the wire coil, the valve of the normally controlled solenoid valve is closed immediately, so the oil circuit of the automobile is shut off. This also plays a certain role in the protection of automobile engines. At the same time, the controller through the second output port controls the second electrode of the electronic switch output trigger pulse signal, the electronic switch is opened, immediately launched the electric putter to push the car brake automatic vehicle brake, to avoid serious traffic accidents happen. The whole process is completed in 10 ms time. Working principle diagram of anti misoperation device for emergency brake is shown in Fig. 3.12.

3.2.2 Anti Misoperation Algorithm for Emergency Braking The core software of this technology is an algorithm that can prevent stepping off the accelerator incorrectly during emergency braking. After the system has configured the key components such as flow measurement and control solenoid valve, it still needs to rely on algorithm software to realize its intelligent work process. The specific steps for preventing the pedal from stepping on the accelerator during emergency braking are as follows. 1. Through the solenoid valve that can be installed on the upper side of the pedal throttle of the automobile oil supply pipeline, the fuel oil differential pressure of the upper and lower valve is transmitted to the pressure input port of the

3.2 Prevent Mistake of Stepping on Accelerator When Stopping Brakes

131

differential pressure sensor. The differential pressure of the fluid is transformed into the differential pressure electrical signal u through the sensing element in the differential pressure sensor, and then outputs to the input port of the signal processor. 2. According to the vehicle model in the signal processor preset brakes mistakenly step on the throttle criterion that is a sudden brake mistakenly steps on the throttle to determine the threshold. Specific methods are as follows. (1) Select the type of vehicle to run on the vehicle dynamic test bench. (2) Set the car at a variety of driving speeds, such as 20, 40, 60 and 80 km/h, etc., and then suddenly sharply pedal the accelerator. Through the fuel flow detection and speed detection device, the transient variation curve of the fuel flow qV under various driving conditions is recorded, and its change rate dqV  is calculated. Among them, u means the speed of the vehicle before dt u mistakenly stepping on the accelerator.   (3) The minimum value of a set of dqdtV  data is determined as the criterion of u sudden brake mistakenly stepping on the accelerator, that is, the brake pedal mistakenly steps on the throttle to determine the threshold value δ. 3. According to the relationship between the fuel volume flow rate qV and the differential pressure p of the upstream and downstream valves qV  k1 p

(3.22)

And the relationship between differential pressure sensor output fluid differential pressure electrical signal u and fluid differential pressure p u  k2 p

(3.23)

Get

qV  k1

√ u k u k2

(3.24)

That is the transient value of fuel volume √ flow. k1 is the ratio coefficient between the volume flow qV and the square root p of the fluid differential pressure, between the electrical signal u and the fluid and k2 is the proportionality factor √ differential pressure p, k  k1 k2 k2 . 4. Signal processor compares the fuel volume flow transient value qV with the fuel volume flow transient value obtained at the previous moment, and calculates the change rate of the fuel volume flow rate at the moment, that is qV dqV ≈ dt t

(3.25)

132

3 Intelligent Prosecution of Dangerous Driving

Among them, qV  qV (tk ) − qV (tk−1 ), qV (ti )(i  0, 1, 2 . . . k − 1, k, . . .) represents the transient numerical value of fuel volume flow at ti moment, t  tk − tk−1 , so there are qV qV (tk ) − qV (tk−1 ) dqV ≈  dt t tk − tk−1

(3.26)

V > δ, the signal processor will immediately identify: the driver mistak5. When q t enly pedals the accelerator at the time of braking, so the signal processor sends brake instructions to the controller. 6. When the controller receives the brake instruction, the trigger pulse is sent to the control pole of the two electronic switches which are connected to the solenoid valve and the electric pusher rod respectively. 7. The control poles of the two electronic switches lead to the source/drain bipolar conduction under the control of the trigger pulse. The process is as follows.

(1) When the first electronic switch is switched on under the control of the trigger pulse, the electric coil of the electromagnetic valve can be controlled to receive electricity, so that the valve in the normally opened state can be closed in time, the transient flow of fuel drops from qV (tk ) to 0 in very short time, the fuel circuit is cut off, therefore, the car engine is closed in time, so that the car lost the driving force ahead, providing the first technical guarantee to avoid traffic accidents. (2) When the second electronic switch is opened under the control of the trigger pulse, the linear motor power of the electric push rod is connected, and the pusher starts the action, the thrust moment of the push rod is transmitted to the brake through the lever of the brake and the hydraulic amplifying mechanism, make the car brake automatically in time, finally, it can overcome the possible collision of vehicle coasting, thus providing the second technical guarantee to avoid traffic accidents. The brake anti misoperation technology has the following beneficial effects. (1) Integration of detection and control of fuel flow, simplify the control system. (2) Through the power electronic switch device, the contactless solenoid valve and the electric push rod can be controlled by the contactless and on/off of the wire coil, and the response is fast. (3) Automation and intelligentization of machine judgment and decision making, V , can the accurate only by calculating the fuel volume flow rate dqdtV  q t decision be made, moreover, the decision threshold δ can be corrected by the actual measurement of vehicle type and its operating conditions, therefore, the technique of the invention can be applied to different types of vehicles. (4) It can significantly improve the level of automation technology and driving safety factor and increase the economic value of its products. According to the measured results, it is proved that the accurate rate of sudden brake can be 100% and the response time is less than 10 ms. The speed of the car is

3.2 Prevent Mistake of Stepping on Accelerator When Stopping Brakes

133

calculated by 80 km/h when the brakes appear, from the automatic closing of the oil path to the automatic braking process, the gliding distance of the car due to hysteresis and inertia is less than 30 cm.

3.3 Monitoring Whether the Driver’s Both Hands off the Steering Wheel Driving a car requires the driver to hold the steering wheel with both hands. The driver must not allow his hands to leave the steering wheel. This is because the driver’s hands are out of the steering wheel in the course of the vehicle, as long as the pavement is not smooth, even a deep pit or an obstacle is found, it is very likely that the vehicle can rush out of the driveway in the vibration, and then have serious traffic accidents, such as the collision with other vehicles. In the current driverless car has not entered the era of utility, to prevent the driver’s hands off the steering wheel drive intelligent inspection technology is clearly an important function of the vehicle should be configured.

3.3.1 Sensing Structure and Pre-processing The core hardware that prevents the driver’s hands from leaving the steering wheel during the driving process is the sensor.

3.3.1.1

Sensor Structure

The core sensor is the structure that is set inside the steering wheel. The structure of the sensor is shown in Fig. 3.13. The inner core of the steering wheel ring contains a soft ferrite ring, three groups of coils are wound in the same direction of the soft ferrite ring, and the head of a group of coils in the three coils relates to the tail end of the other group of coils in series to form a series inductor. The two ends of the total inductance of the three groups of coils are connected by wire to other electrical components to form a LC oscillation loop. The outer layer is covered with a shell, the shell can be made of synthetic plastic or wood material. The sensor structure of the steering wheel is called the steering wheel handshake sensor, referred to as the steering wheel sensor.

134

3 Intelligent Prosecution of Dangerous Driving

Inductance coil

Housing

Ferrite Fig. 3.13 Steering wheel sensing structure

VCC

Rf C f T1

T2

Signal processor

Controller

Voice player

Steering wheel sensor

Fig. 3.14 Sensor signal processing circuit

3.3.1.2

Front Signal Processing Circuit

The signal output of the steering wheel sensor needs to be pre-processed by the front circuit. The pre-processing circuit consists of LC oscillation, high frequency amplification, limiting and demodulation circuits (see Fig. 3.14). In Fig. 3.14, the T1 and LC circuits assume high frequency signal amplification while constituting an oscillating circuit, the Rf and Cf are positive feedback resistors and capacitors in the oscillating circuit, by fine-tuning the Rf , it can ensure the stability of the oscillation circuit. T2 can ensure the amplitude limiting of the high frequency signal by setting the correct quiescent operating point.

3.3.2 Sensing and Control Principle The three groups of coils wound on the surface of the soft ferrite surface of the steering wheel are connected in series to form an electromagnetic inductor. When holding the steering wheel with hands or one hand, the inductance of the electromagnetic sensor will change because of the electrostatic effect.

3.3 Monitoring Whether the Driver’s Both Hands off the Steering Wheel

135

After debugging, the system is in the resonant state, that’s why, when both hands (or one hand) does not hold the steering wheel, the oscillating circuit is in the resonant state, and the high frequency oscillation signal is amplified by the high frequency to reach the maximum potential value of the limiter. The fluctuation of the operating voltage has been considered in the design, and the resonant circuit has a certain bandwidth.

3.3.2.1

Initialization

In view of the automobile battery voltage attenuation and fluctuation, as well as the randomness of the surrounding environment interference factors, before the formal operation of the system must be initialized, that is know as “system learning”. The initialization steps are as follows. (1) Voice player automatically prompts: “please leave your hands off the steering wheel”, after the delay of 2 s, each interval of 10 ms, the signal processor automatically collect a signal. The DC voltage collected each time is stored in the first array unit after analog to digital conversion. Collect 10 times, complete data acquisition within 100 ms, form the first sampling voltage vector V(1) . That is   (3.27) V(1)  V1(1) V2(1) . . . VM(1) In the formula, M is the number of samples, such as M  10. (2) Voice player automatically prompts: “please use Left hand to hold the steering wheel”, after the delay 2 s, each interval of 10 ms, the signal processor automatically collects a signal. The DC voltage collected each time is stored in the second array unit after analog to digital conversion. Collect 10 times, complete data acquisition within 100 ms, form second sampling voltage vectors V(2) , that is   (3.28) V(2)  V1(2) V2(2) . . . VL(2) In the formula, L is the number of samples, such as L  10. (3) Voice player automatically prompts: “please use right hand to hold the steering wheel”, after the delay 2 s, each interval of 10 ms, the signal processor automatically collects the first signal. The DC voltage collected each time is stored in the third array unit after analog to digital conversion. Collect 10 times, complete data acquisition within 100 ms, form third sampling voltage vectors V(3) , that is   (3.29) V(3)  V1(3) V2(3) . . . VR(3)

136

3 Intelligent Prosecution of Dangerous Driving

In the formula, R is the number of samples, such as R  10. (4) Calculate the mathematical mean V¯ (1) of the first sampling voltage vector V(1) element M 1 (1) V¯ (1)  V M i1 i

(3.30)

(5) Calculate the mathematical mean V¯ (2+3) of second sampling voltage vector V(2) and third sampling voltage vector V(3) elements

L ¯ (2+3)

V



j1

Vj(2) +

R k1

Vk(3)

L+R

(6) Determine the threshold δ1 and δ2     δ1  maxVi(1) − V¯ (1)  i  1, 2, . . . , M ;          δ2  max maxVj(2) − V¯ (2+3) , maxVk(3) − V¯ (2+3)  j  1, 2, . . . , L k  1, 2, . . . , R 3.3.2.2

(3.31)

(3.32)

(3.33)

On-Line Real-Time Monitoring

After initialization, the system enters the online real-time detection state as the vehicle runs. (1) When the signal processor receives the signal voltage Vt at a certain time t, if  Vt − V¯ (2+3)  > δ2 , the driver is normally driving, that is, holding the steering wheel with both hands or one hand for driving control, this is because the oscillation loop is far away from the resonance point, so that the value of Vt is small, which is caused by the deviation from V¯ , therefore, the system does not interfere with the operation of the driver. (2) When the signal voltage Vt at a certain time t, if   the signal processor receives the Vt − V¯ (2+3)  ≤ δ2 and Vt − V¯ (1)  > δ1 , the driver’s single hand holding the steering wheel is more relaxed, the system gets alert and will prompt: “hold the steering wheel, please!” (3) When the signal processor  receives    the signal voltage Vt at a certain time t, if Vt − V¯ (2+3)  ≤ δ2 and Vt − V¯ (1)  ≤ δ1 , the driver’s hands are already detached from the steering wheel, that is, “double take off”, the system immediately sends a warning through the voice player: “hold the steering wheel, and never leave the steering wheel with your hands!”, at the same time, the vehicle is forced to intervene by the controller, such as deceleration or braking.

3.4 Intelligent Recognition Technology for Driver’s Fatigue Driving

137

3.4 Intelligent Recognition Technology for Driver’s Fatigue Driving The so-called “driver fatigue driving” is that the driver is extremely tired when driving. In other words, its main performance is the driver’s lethargy due to fatigue. This is a frequent occurrence for long distance truck drivers. According to the US National Sleep Foundation report, there are more than 100 thousand cases of traffic accidents caused by driver fatigue in the United States each year. The analysis causes of many of traffic accidents shows that the driver’s perceived fatigue and decision-making fatigue are the main causes of driving accidents in driving fatigue. The study of more than 38000 causes of accidents in Japan showed that perceived fatigue accounted for 40.1%, and decision-making fatigue was 41.5%. Driving fatigue affects driver’s alertness and safe driving ability. Driver fatigue has attracted worldwide attention, the western developed countries have invested huge manpower and material resources to carry out the research work of driving fatigue. In China, the backward reality of driving fatigue monitoring method and severe road traffic safety situation have urged people to solve the difficult problems in driving fatigue monitoring technology. At present, the research of driver fatigue identification technology mainly focuses on the following three aspects. (1) Monitoring methods based on the individual characteristics of the driver, such as eyelid activity, eye closure, nodding movements, etc. (2) Monitoring methods based on physiological parameters of drivers, such as electroencephalogram, eye movement, electromyography, muscle activity, etc. (3) Monitoring methods based on vehicle parameters, such as speed, acceleration, vehicle position, etc. To overcome the driver fatigue phenomenon, in addition to strengthening the legal education of driver necessary traffic and transportation enterprises production management, relying on advanced technology to improve the existing vehicle safety driving intelligent function is important technical means.

3.4.1 Hardware Configuration The hardware system of an intelligent identification of driver fatigue state includes pinhole camera, signal processor, controller and voice player. A pinhole camera used to identify driver fatigue states is installed in the driver’s cab, facing the driver’s face, as shown in Fig. 3.15. The composition of the signal processor is like the other identification techniques. After processing and analyzing the signal of the driver’s face, the signal processor

138

3 Intelligent Prosecution of Dangerous Driving

Fig. 3.15 Position of pinhole camera for identifying driver’s fatigue state

Pinhole camera

determines the output of the control command. The controller controls the throttle and the brake of the car according to the control instructions. According to the result of recognition, voice player warms to remind the driver should stop to rest.

3.4.2 Core Algorithm The basic idea of identifying driver’s fatigue state is a core algorithm. Track the driver’s eye expression, the driver closed eyes once the time is more than 3 s, the intelligent identification system is that the driver appeared tired, fatigue driving state, the continuation of technical means must be taken to stop the situation. The specific steps of the algorithm include training sample learning, eye region tracking, eye state test sample collection, feature extraction and recognition, recognition results command output.

3.4.2.1

Training Sample Learning

1. Collect training image As shown in Fig. 3.16, a sufficient number of samples of the face and eye images for drowsiness caused by fatigue are collected to form a driver fatigue state image training sample set, in which each image has 256 gray levels. 2. Convert the image into one dimensional vector Each two-dimensional eye image data is transformed into a one-dimensional vector, and the “drowsiness” is defined as the 1 type of eye features, “non-sleepy” is -1 type of eye characteristics. Therefore, one dimensional vector xi of the i image can be expressed

3.4 Intelligent Recognition Technology for Driver’s Fatigue Driving

139

Fig. 3.16 Driver fatigue state image training sample set legend

T  T  xi  xi1 xi2 . . . xim  xij

(3.34)

In the formula, xij is the gray value of the j pixel of i sample of 1 class. i  1, 2, . . . , n is the 1 type of eye samples, here n  9, so i  1, 2, . . . , 9. j  1, 2, . . . , m for each sample image taken as prime number, m  u × v,u and v are the sample image column prime and row image prime number, respectively. When each sample image is u  246, v  112, then m  27552, at this point formula (3.34) can be expressed as T  T  xi  xi1 xi2 . . . xi27552  xij

(3.35)

3. Calculation of eigenvalues and eigenvectors of training samples Calculate the mean x¯ of the 1 kind 1 xij n × m i1 j1 n

x¯ 

m

(3.36)

The resulting mean is called the 1 class of eye averaged images. The above training samples can be expressed as standardized vi  xi − x¯ ; i  1, 2, . . . , n

(3.37)

The 1 types of eye normalization vectors v are composed of training samples  T v  v1 v2 . . . vn

(3.38)

At the same time, the covariance matrix of the 1 eye images is  T   v1 v2 . . . vn ; Q ∈ Rn×n Q  v1 v2 . . . vn

(3.39)

140

3 Intelligent Prosecution of Dangerous Driving

Using the formula (3.39), we obtain the eigenvalue λl of Q and its feature vector, and rearrange it from large to small to get feature vector as follows T  p  λ1 λ2 λ3 . . .

(3.40)

Among them, λ1 ≥ λ2 ≥ λ3 ≥ . . . 4. The training samples are transformed into feature space by linear transformation Because the feature vectors corresponding to the larger eigenvalues contain more feature information in the human eye region, therefore, we can select the vector space of the feature vectors corresponding to the larger eigenvalues of the front s, which can approximate the main information of the human eye image. T  The n images xi  xi1 xi2 . . . xim (i  1, 2, . . . , n) in the image library can T  be projected to the feature space, and the projection vector i  ωi1 ωi2 . . . ωim is obtained. T  From the v  v1 v2 . . . vn , the normalized values corresponding to the larger eigenvalues of the first N are selected to form the new normalized vectors  T vˆ  v1 v2 . . . vs

(3.41)

That’s why, after establishing the normalized vector of the facial eye area feature, it can be used as a criterion to identify whether the driver is drowsy because of fatigue.

3.4.2.2

On-Line Real-Time Identification Method

1. Image enhancement The real time image collected by pinhole camera in the cab is enhanced. PulseCoupled Neural Networks (PCNN) is used to enhance the image. The so-called PCNN is a link model that simulates the synchronization of neurons associated with characteristics to demonstrate the phenomenon of pulse propagation. Therefore, it has a natural connection with the perceptual ability of the visual nervous system. In the PCNN structure model applied to image processing, each pixel f (i, j) of the image to be processed corresponds to each neuron Nij . The pixel coordinates, i  1, 2, 3, . . ., j  1, 2, 3, . . . The pixel intensity value of the pixel f (i, j) is represented by Iij , each neuron Nij receives Iij from outside stimuli, the feed input Fij (t) and the join input Lij (t) are also received from other neurons in the internal network. Then, the internal behavior Uij(t) of neurons  Nij is formed by the coupling strength β of neurons in the form Fij (t) 1 + βLij (t) of product coupling. The output of Yij (t) (also known as ignition) is stimulated or suppressed by the comparison of dynamic threshold θij (t) with Uij (t). The t stands for time.

3.4 Intelligent Recognition Technology for Driver’s Fatigue Driving

141

PCNN discrete mathematical model is Fij (n)  e−αF Fij (n − 1) + vF mijkl Ykl (n − 1) + Iij

(3.42)

Lij (n)  e−αL Lij (n − 1) + vL wLijkl Ykl (n − 1)

(3.43)

  Uij (n)  Fij (n) 1 + βLij (n)

(3.44)

 Yij (n) 

1, Uij (n) > θij (n) 0, Uij (n) ≤ θij (n)

θij (n)  e−αθ θij (n − 1) + vθ Yij (n)

(3.45)

(3.46)

In the formula, mijkl and wijkl are the elements of the inner connection matrix M and W of the neural network, respectively. Connection weights between neurons in the feeding domain and the join domain. That is, the synaptic gain strength between the feeding domain and the link domain between the l neuron Nijl and the k neuron Nijk . αF and αL are attenuation time constants of feeding domain and link domain between l neuron Nijl and k neuron Nijk , respectively. αθ is the decay time constant of the dynamic threshold function. vF , vL and vθ are the amplification coefficients of the feeding domain, the coupling domain and the threshold output, respectively. The n represents the ignition discrete time of neuron Nij (n  1, 2, 3, . . .). It is easy to see through the analysis of the above mathematical model, the firing output Yij of each neuron not only leads to the change of θij , Lij and Fij output, but also affects the internal behavior and output of other neurons adjacent to it. In turn, the output changes of these affected neurons also stimulate the internal behavior and output of the neighboring neurons adjacent to them and so on, within the PCNN formed a widely spread information channel. Due to the intensity difference between the pixels on both sides of the edge of the conventional image is greater than that of the intensity of the pixels adjacent to the space in the region, the intensity difference is relatively large, therefore, if PCNN is used in two-dimensional image processing, each neuron corresponds to the pixel of the image, the intensity of the intensity is the external stimulus of the neuron, and in the interior of the PCNN, the spatial neighborhood and intensity similar cluster can synchronize the ignition, otherwise asynchronous ignition. In the image enhancement, the image pixels corresponding to the synchronous ignition exhibit the same brightness intensity value, thus smoothing the image region. Asynchronous image pixels corresponding to the ignition strength of different show the brightness value, thereby increasing the brightness gradient intensity of image regions, and highlighted the edge of the image, the image brightness intensity distribution after enhancement is more hierarchical.

142

3 Intelligent Prosecution of Dangerous Driving

In the standard PCNN model, the output is a two-valued image frame due to the function of hard clipping function. To make the output of the PCNN mapping function proposed can effectively handle the image’s global contrast enhancement, human visual perception characteristics based on the above, using a logarithmic mapping function, the image brightness is mapped to a suitable visual range. The mapping function of PCNN output is Yij (n)  ln Imax − t(n − 1)/τθ

(3.47)

In the formula, Imax is the intensity value of the brightest pixels in the original image. t is the sampling time cycle, that is, the iteration time step. τθ is the time constant, and t(n−1)/τθ represents the decay step of the dynamic threshold function in the (n − 1) ignition time in PCNN. Yij (n) is the perception output of neuron Nij at n ignition time, that is to enhance the brightness intensity of the image. The greatest advantage of this method is that it has a natural connection with the perceptual ability of the visual nervous system, the model not only can smooth the image area and highlight the image edge, but also can improve the visual effect of color image and enhance the true effect of image color. 2. Color space conversion The enhanced image is transformed into a color space, that is, the enhanced digital image is converted from the RGB color space to the HSV color space, the converted hue H, saturation S and brightness V are expressed as V ≤ max(R, G, B)

(3.48)

(3.49)

(3.50)

... ◦ ... And in the process of calculating H, if H < 0 appears, then take H ≤ H + 360 , H is the actual value of H.

3.4 Intelligent Recognition Technology for Driver’s Fatigue Driving

143

3. Driver eye area tracking (1) In the collected face images, the scene color hue of the detected pixel is scanned according to the coordinate sequence from left to right and fromtop to bottom, ◦ ◦ the pixels corresponding to the color hue set of the hue interval 2 , 47 in the HSV color space are formulated as the face region, therefore, the other areas of the characters and their image background can be distinguished from the human face pixels accurately. This is because there is an obvious difference in hue between the face color and the clothes and background in the character images, color hue of facial complexion, clothing and background is distributed in different angle regions, moreover, the hue of the face color is concentrated stably in a certain angle region in the HSV color space. It is proved by experimentation that the angle distribution of skin tone H is basically ◦ maintained between 2 ∼ 47 of HSV color space, whether it is natural illumination or artificial light source, or whether the camera system is similar or not, therefore, the facial color, clothing, background and other scenery can be distinguished by the hue value of the human image in the HSV space. In other words, only when the hue of a ◦ ◦ scene is within the range of 2 , 47 , it is possible to confirm that it is the color of the face, otherwise these are other objects, such as clothing or other objects. It is further ◦ proved by experiment that the probability of 11 of face color is the highest, so the ◦ peak of face color. The distribution probability of hue value of 11 is the probability  ◦ ◦ ◦ face color hue in interval 2 , 47 is P(H ), the probability of H  11 is the highest, ◦ ◦ that is, P 11  Pmax , that is, when the hue of a scene is 11 , the confidence of the face color is highest. (2) To determine the center of the face and the eye region In the face color tone set, the pixel coordinates corresponding to the hue values ◦ closest to 11 are used as the face points. For example, after face area  ◦ center ◦ that falls into the HSV color space is , 47 search results, the hue set of the 2   ◦ ◦ ◦ . . . , 9.7 , 10.1 , 9.5 , . . . , and the pixel coordinates corresponding to the set are ◦ ◦ {. . . , (ik−1 , jk−1 ), (ik , jk ), (ik+1 , jk+1 ), . . .}. The closest hue to 11 is 10.1 , the corresponding pixel coordinates are (ik , jk ), therefore, (ik , jk ) can be determined as the face center position coordinate, i represents the column coordinates of pixels, j represents the row coordinates of pixels, the number of columns and number of rows are represented by foot labels, The k of ik stands for column k, the k of jk stands for line k. The eye tracking region of u × v can be obtained by expanding the pixels of u rows and expanding the pixels of 2v columns to both sides according to the center of the face. (3) Eye region tracking of human face The first order prediction algorithm is used as the eye region tracking method of human face. T  The speed of the facial motion is V (tk )  Vi (tk ) Vj (tk ) , and

144

3 Intelligent Prosecution of Dangerous Driving



Vi (tk ) Vj (tk )



⎡ ⎢ ⎣

ik −ik−1 tk −tk−1 jk −jk−1 tk −tk−1

⎤ ⎥ ⎦

(3.51)

That’s why, when the interval time is t, the two operations of the center position of the face are calculated, t  tk − tk−1 The first order predictive estimate should be the following ⎤ ⎡ i −i ⎤ ⎡ k−1 k−2 V˜ (t ) tk −tk−1 ⎥ ⎣ i k ⎦⎢ (3.52) ⎣ j −j ⎦ k−1 k−2 V˜j (tk ) t −t k

k−1

The pixel coordinates of the target are predicted ⎡ ⎤ ⎡ ⎤   ˆik+1 V˜i (tk )(tk − tk−1 ) i ⎣ ⎦⎣ ⎦+ k jk ˆjk+1 V˜j (tk )(tk − tk−1 )

(3.53)

In the formula, Vi (tk ) and Vj (tk ) are the components of the two coordinate axes (i and j) of the velocity V (tk ) at the moment k in the pixel coordinate system. V˜i (tk ) and V˜j (tk ) are the estimated components of the velocity V (tk ) at the two coordinate axes (i and j) at the moment k. ik , ik−1 and ik−2 are the i coordinate values at the k, k − 1 and k − 2 moments respectively. jk ,jk−1 and jk−2 are the j coordinate values at the k, k − 1 and k − 2 moments respectively. ˆik+1 and ˆjk+1 are coordinate estimates of i and j at k + 1 moment, respectively. Therefore, when the face center is (ik , jk ) at moment tk , it can  pass the first order prediction algorithm, predict that the center of human face is ˆik+1 , ˆjk+1 at moment tk+1 . 4. Import test samples According to the face center coordinates (ik , jk ) tracked by the recognition system, the pixels of u rows are expanded upwards, and the pixels of 2v columns are expanded to both sides, and the u × v eye region image is intercepted as the import test sample. 5. Computing the image feature vector of the sample to be tested According to the same method of training samples, the image feature values and eigenvectors of the test samples are calculated. The test sample is projected into the feature subspace. At this point, the image of the eye region corresponds to a point in the feature subspace. 6. Recognition of driver’s fatigue state The u × v eye area test image projected to the subspace is compared with the training image one by one to determine the category of the sample to be identified. Distance classification function can be used to identify. When

3.4 Intelligent Recognition Technology for Driver’s Fatigue Driving

!   ! G v˜ , vˆ  !v˜ − vˆ ! ≤ ε

145

(3.54)

Explanation: at present, the driver’s fatigue state is classified into 1 category, that’s why, the driver is in a state of fatigue. Otherwise, the current “determine driver fatigue state” belongs to the -1 categories, that’s why, determine the driver is in the normal driving state. In formula (3.54), v˜ and vˆ represent test samples and training sample feature subspaces respectively. 7. Control decision output According to the recognition results, the control command output is determined for the current driving condition. That is, once the driver is in a state of fatigue, the system outputs the control command in real time, to remind that the driver should stop to rest, according to the speed driven by the optimized control strategy it makes the car gradually slow down and eventually stop.

Chapter 4

Intelligent Monitoring Technology of Automobile Power and Transmission System

The intelligent monitoring technology of automobile power and transmission system started from the early electronic control technology. The development of electronic control technology has roughly gone through three stages. The first stage (1965–1975), a simple electronic device for engine electronic ignition. The second stage (1975–1985), such as an engine electronic control system, anti-lock braking system and some other independent control system. The third stage (after 1985), the powertrain integrated engine and transmission system control system, the traction control system of integrated power and brake system, as well as the integrated control system of vehicle environment, etc. With the continuous development of the engine, the control parameters and control strategies are constantly updated with the continuous improvement of the engine. However, in view of the intelligent control theory and technology of automobile engine, it must be expounded through professional literature that’s not what this book can do. Therefore, this chapter will focus on some intelligent monitoring technologies and methods which are beyond the control of power systems.

4.1 Intelligent Control of Automobile Engine and Transmission System The control of engine mainly includes fuel injection control, ignition timing control, knock control, idle speed control, exhaust gas recirculation control and air-fuel ratio closed loop control. All of these are scientific research projects that can be properly solved by special research. From engineering control point of view, these topics are all around the dynamic characteristics of the engine. In other words, the optimal control of engine operation is to consider the dynamic characteristics of vehicle power and transmission system. Therefore, integrated control of a vehicle engine and transmission system is a must to consider. © Shanghai Jiao Tong University Press, Shanghai and Springer Nature Singapore Pte Ltd. 2019 X. Zhang and M. M. Khan, Principles of Intelligent Automobiles, https://doi.org/10.1007/978-981-13-2484-0_4

147

148

4 Intelligent Monitoring Technology of Automobile Power …

4.1.1 Basic Functions of Power Control Systems (1) Parameter acquisition Automotive power train control system is a very complex system, with strong realtime, nonlinear and uncertainty, so to meet the needs of the operational control of the complex, it needs many parameters related to signal acquisition. Mainly includes speed, engine speed, throttle opening (or fuel flow), clutch travel and its current position and other signals. Acquisition of dynamic parameters of a power system is the basic condition to realize control. There are many kinds of sensor technology that depends on the parameter acquisition, and different sensing methods which will directly affect the control effect of the system. For example, the throttle opening, the purpose of testing this parameter is to master the fuel flow, by measuring the throttle opening. It is an indirect measure to know the fuel flow. Moreover, the measurement of the opening depends on the displacement (angle) sensor, and it depends on the machining accuracy of the sensor, more importantly, from the opening conversion to the flow process, it is also constrained by the oil pressure parameters. Only when the upstream pressure of the throttle is constant, the opening is the only factor to determine the flow, but in fact it can not do this. Therefore, by detecting the throttle opening method to indirectly grasp the fuel flow error is great and random, so that the “air-fuel ratio” control objectives and optimization have become impossible. At present, most cars still use the throttle opening as an indirect measurement method of fuel flow. If you can use the above “sudden brake anti mistake stepping” technology mentioned in the “flow measurement and control solenoid valve” directly measure fuel flow, “air-fuel ratio” optimization control objectives can be truly realized. (2) Real time processing and analysis of dynamic parameters The analysis and processing of the dynamic parameters of the engine are borne by the vehicle signal processor. The process of analysis and processing is based on the input dynamic parameters of the engine, and finally determines the control mode of the engine, clutch and transmission. The system first transfers the signal collected by the sensor to the signal processor through a conditioning circuit. The input parameter signal is processed and analyzed by the operation program (algorithm) in the signal processor, and the corresponding control signal is finally output to the actuator. (3) Driving the actuating mechanism The control command of decision result is converted into analog voltage signal by controller through signal processor, the analog voltage signal is amplified by the drive circuit to drive the actuator to adjust the controlled parameters, at the same time, the change value of the controlled parameter is fed back to the signal processor through the sensor, finally, the control of automobile power and transmission system is achieved. The operation program (algorithm) in the signal processor preset is an important technology of intelligent control.

4.1 Intelligent Control of Automobile Engine and Transmission System [Dynamic parameter detection subsystem] Speed signal Engine speed signal Fuel flow signal Transmission lever position signal Brake switch signal Driving mode signal

Mathematical model control program

149

Input interface

Conditioning circuit

Arithmetic device

Reference model and its parameter database

Safety circuit

Output interface

Mutual exchange of external signals

[Executive body] Shift control Shift quality control Oil pressure control Torque control Heat exchange flow control

Fig. 4.1 Principle block diagram of vehicle control system

4.1.2 Basic Structure of Power Control System The power control system functions change according to the intention of driver and vehicle driving environment, automatically adjust the transmission component ratio and operating parameters, to achieve the optimal overall performance of power and transmission efficiency and the best vehicle. Usually, the vehicle control system is mainly composed of three parts of the vehicle data acquisition system (sensing system), signal processor and executive mechanism. As shown in Fig. 4.1, the signal processor mainly includes: sensing signal input interface, conditioning circuit, mathematical model and control program, reference model and its parameter database, arithmetic unit, safety circuit and output interface, etc. Automotive power and transmission control is a real-time multi task control system. The development trend of the control system is to simplify the hardware structure and enrich the software function, to improve the system reliability and intelligence, and to facilitate the expansion of the system function. Figure 4.2 shows a representative system master control flow.

150

4 Intelligent Monitoring Technology of Automobile Power … Initialization

Parameter detection input

System state recognition

Other auxiliary control

Engine control

Transmission control

Fig. 4.2 Main control block diagram of the system Automobile power transmission system

Parameter adaptation and fault diagnosis

Oil valve coordination

Engine target speed

Oil valve opening

Parameter detection 1

Organizational level

Clutch coordination

Clutch control

Clutch engagement stroke

Coordination level

Control level

Parameter detection

Fig. 4.3 Three-stage hierarchical control structure framework for automobile power

4.1.3 Three-Level Hierarchical Intelligent Control It is difficult to establish an accurate control system model because of the variability of automobile power and transmission system and the nonlinearity of the system. Therefore, it brings some difficulties to the control method. Three-level hierarchical control is an intelligent strategy to realize coordinated control between clutch and throttle in the stage of vehicle restoring power. The so-called “three-level”, that is, the entire intelligent control by the organization level, coordination level and control level three components, the structure of the framework as shown in Fig. 4.3.

4.1 Intelligent Control of Automobile Engine and Transmission System Gear

Engine speed

Engine target speed

Speed difference between clutch master and slave

Clutch control knowledge base

Speed

151

Inference structure of clutch control Actual speed and target speed difference

Accelerator pedal

Clutch control law

Throttle control law

Throttle control inference mechanism

Throttle control knowledge base

Fig. 4.4 Block diagram of organization level

4.1.3.1

Organization Level Work Principle

Organization level is the core and key of clutch and engine integrated intelligent control during clutch engagement. The organization level is to determine the basic control law of the throttle and clutch under the working condition according to the current vehicle operating parameters and the mathematical model that defines the relationship between the throttle and clutch. At the same time, this level also has the learning function that can modify the parameter control relationship in the knowledge base according to the adaptation degree of the vehicle operating condition parameters of the coordination level feedback, so it can continuously improve the control quality. Fault diagnosis is based on the running state and dynamic characteristics of the system. The system’s operating conditions are comprehensively evaluated, once the exception occurs, during real-time diagnosis and processing. The organization level structure is shown in Fig. 4.4. Order: the engine speed is neo , the front and back gear ratios are Iga and Igb respectively. If the vehicle speed is assumed to be constant before and after gear shifting, the starting motive speed ned is calculated. ned 

Igb neo Iga

(4.1)

In the throttle control knowledge base, the control relationship between the acceleration pedal stroke, the actual engine speed and the target speed difference, and the clutch control law and the throttle opening are stored. The clutch control knowledge base store gear, accelerator pedal stroke, clutch main and driven plate speed difference, etc., and the relationship between the clutch engagement speed and the clutch

152

4 Intelligent Monitoring Technology of Automobile Power …

< − 50r min

Actual and target speed difference

Automobile engine

> 50r min Speed up clutch engagement

Increase the throttle opening

Greater than the rated value?

No

No

Greater than the rated value?

Yes Loosen clutch engagement

Yes Reduce throttle opening

Fig. 4.5 Coordination flow chart

engagement speed. The clutch engagement speed is the main control parameter to determine the throttle opening. There is an increasing relationship between them, that is, with the engagement of the clutch, the throttle opening will increase accordingly. Engine actual speed and target speed difference are auxiliary parameters of throttle control. Throttle opening is a comprehensive index to reflect the engine running state and driver’s intention and is also the main parameter to determine the clutch engagement speed. The speed difference between the main clutch and the driven plate is the auxiliary control parameter of the clutch engagement speed. Because of the complexity of automobile power transmission system operating conditions, changeable, nonlinearity and uncertainty, it is difficult to establish a precise mathematical model of the relationship between control, so in the clutch and throttle control inference mechanism, according to the knowledge and experience in the vehicle information, can use the reasoning mechanism of fuzzy control.

4.1.3.2

Work Principle of Coordination Level

The main task of the coordination level is to coordinate the action of each controller, determine what conditions it needs to adjust its corresponding control strategy, and select which kind of dynamic compensation. The task of the coordination level can coordinate the throttle and clutch control with the engine target speed ned as the control target, and the coordination process is shown in Fig. 4.5.

4.1 Intelligent Control of Automobile Engine and Transmission System

< − 50r min

Actual and target speed difference

153

Engine

Parameter fuzzification

Fuzzy decision

Defuzzification

Fuzzy rule base

Throttle opening

...

Fig. 4.6 Flow chart of throttle fuzzy control

4.1.3.3

Working Principle of Control Level

The control level is mainly based on the control parameters determined by the coordination stage, and according to the characteristics of the control object, the effective control algorithm is used to complete the task of the throttle and clutch. 1. Throttle Control In view of the complex structure of the engine, the nonlinear relationship between throttle opening, load and speed, there exist many of interference and uncertainties in operation. Therefore, the fuzzy control method is adopted for engine control. The basic principle is shown in Fig. 4.6. The control of the engine is mainly the problem of the recovery of oil supply after the starting of the vehicle. The engine is to maintain its constant speed by constantly adjusting the throttle opening to overcome the change of the load. According to the analysis of the organization level, we can see that the recovery size of the engine oil supply is mainly determined by the joint position of the clutch and the deviation between the actual engine speed and the target speed. Order: fuzzy set corresponding to the domain of deviation between engine actual and target speed is {Negative big, Negative middle, Negative small, Negative zero, Positive zero, Positive small, Positive median, Positive big} = {NB, NM, NS, NO, PO, PS, PM, PB}

(4.2)

The fuzzy set corresponding to the deviation region of clutch engagement position is {Positive zero, Positive small, Positive median, Positive Big} = {PO, PS, PM, PB} (4.3)

154

4 Intelligent Monitoring Technology of Automobile Power …

Table 4.1 Membership degree table of control quantity Clutch Speed deviation position PO PS PM PB

NB

NM

NS

NO

PO

PS

PM

PB

1 −1 −6 −5

2 0 −5 −4

3 1 −2 −2

4 3 −1 0

4 4 0 0

5 4 2 2

6 5 4 3

6 5 5 4

Knowledge base The relationship between motion parameters and intention

Parameter fuzzification

Shift decision-making knowledge

Feature recognition inference engine Fuzzy inference engine

Sensing parameters

Shift decision-making reasoning machine

Defuzzification

Transmission

Fig. 4.7 Basic structure block diagram of automatic shift fuzzy controller

Then, the control quantity membership degree value is shown in Table 4.1. 2. Clutch Control Clutch control has always been the focus and difficulty of the integrated control of automobile power and transmission system. The basic structure of the automatic shift fuzzy controller is shown in Fig. 4.7. Vehicle state and driver’s driving intentions are one of the most important fuzzy reasoning objects. In the process of vehicle exercise, the technology that mostly reflects the driver’s intention and the driver’s mastery is to control the opening of the throttle, the speed and acceleration of the vehicle are controlled mostly to reflect the operating conditions and environmental conditions of the vehicle, but in order to change the speed of the vehicle, up or down is an effective way for the driver to realize his intention. In the fuzzy control of shift law, the speed and acceleration of throttle opening, reflecting vehicle operation and environmental conditions are used as shift control signals, and the location of the gears is output. In order to achieve the above objectives, the database of knowledge based on the throttle opening and vehicle speed and the acceleration of the vehicle must be first used. The shift fuzzy control system adopts a multi input and single output structure. According to the vehicle speed, acceleration and throttle opening size, and according

4.1 Intelligent Control of Automobile Engine and Transmission System

155

to the certain fuzzy control rules and reasoning method, the gear shift decision is carried out. Order: the fuzzy quantity of vehicle speed is {Small, Medium small, Medium, Medium large, Large} = {S, MS, M, ML, L} (4.4) The fuzzy quantity of acceleration is {Negative large, Negative small, Zero, Positive small, Positive large} = {NL, NS, O, PS, PL}

(4.5)

The fuzzy quantity of throttle opening is {Small, Medium small, Medium large, large} = {S, MS, M, ML, L}

(4.6)

The shift is represented by a single point, the values are I, II, III and IV, respectively. The fuzzy input space obtained by the fuzzy segmentation of the input parameters constitutes the database of the fuzzy control knowledge base. In shifting fuzzy control, the expert experience system (control rule base) is used to provide decisionmaking for shift control. This expert system (or the experience system) is made up of many of control laws based on many of drivers’ driving experience. It can be said that it is the optimal shift control empirical base. If this expert system can cover all the possible situations that may occur during the shift process, it is fully competent for the automatic shift task. However, as the expert system is too large, it will lead to the slow speed of fuzzy reasoning, and the complexity of the expert system will cause the control to become too sensitive. Therefore, the expert system of shift control can not be too complicated. Generally, there are more than ten to dozens of control rules, so long as we can express the control requirements and satisfy the completeness of control rules, we can express it comprehensively.

4.2 Intelligent Identification Technology of Power and Transmission System With the continuous expansion of the scale of the highway construction and the increasing mileage, the traffic safety of the freeway has been paid more and more attention. Among them, traffic accidents on motorways caused by mechanical failure of cars are common. Especially when the vehicle speed is as high as 120 km or more, once the failure of the automobile engine and transmission system fails to be detected in time, it will easily affect the normal operation of the expressway, and there may be some vehicle damage and casualties. To ensure the safety of the car on the highway, it is necessary to maintain the vehicle in advance. More importantly, the car itself should have the real-time detection function of the fault. In this way, the accident

156

4 Intelligent Monitoring Technology of Automobile Power …

can be prevented. However, how to monitor the working state of automobile engine and transmission system in real time is not easy. The technical difficulties are: (1) With the improvement of automobile sealing performance, the abnormal sound from automobile engine and transmission system is not easy to be detected by drivers and passengers. (2) In the course of driving, drivers and passengers may lose their vigilance because of their ignorance of their own knowledge and experience, even if they hear the abnormal sound of the automobile engine and transmission mechanism. (3) Although the more sophisticated cars have set up a wide variety of sensors and testing instruments, however, due to the limitation of current technology, besides speed, torque, temperature, hydraulic and other conventional parameters can be detected in real time, it is impossible to detect directly or indirectly a lot of unconventional operation parameters. For example, the fatigue of the driving shaft, the fatigue and fracture of fastening screws, etc. Therefore, to find a new technology for real-time detection of vehicle dynamic fault is related to the safety of vehicle operation.

4.2.1 Identification Device Intelligent identification device for abnormal state of power and transmission system includes: pickup sensor, signal conditioner, signal processor and voice player. The pickup sensor is also called sound sensor or voice sensor, referred to as the pickup. The schematic diagram of the system is shown in Fig. 4.8. In the recognition system, the first pickup set in the engine frame, is used for receiving sound signal of engine operation. Second pickup arranged in the back seat of the car frame, can be used for receiving sound signals emitted by the vehicle frame. Figure 4.9 shows a pickup position setting in the car. The signal processor in the identification device includes input interface, A/D converter, operation module, memory module and output interface. The structure principle is shown in Fig. 4.10. Among them, the operation module is responsible for the processing, analysis and judgment of the received sound frequency signals. Memory module stores sample feature database. First pickup sensor Signal conditioner 1 Second pickup sensor Signal conditioner 2

Signal processor

Voice player

Controller

Speed control or brake

Fig. 4.8 Principle block diagram of sound recognition device in power transmission system

4.2 Intelligent Identification Technology of Power and Transmission …

157

Fig. 4.9 Schematic diagram of pickup settings

Pickup sensor

Input interface

A/D

Operation module

Output interface

Memory module

Fig. 4.10 Schematic diagram of signal processor

The controller in the device includes the D/A converter and the voltage amplifier. After the decision instruction output from the operator is transported to the controller, the D/A converter converts it into an analog signal. When the analog voltage signal is amplified by a voltage amplifier, it is used to drive the corresponding servomechanism to achieve control. The basic working process of the intelligent identification device for the abnormal state of power and transmission system is composed of the following steps. (1) The first and second sound sensors are used to detect the sound signals of the automobile engine and the transmission mechanism in real time. (2) The sound signals detected by the two sound sensors are amplified by the conditioner respectively, and then the amplified signals are sent to the signal processor. (3) The signal processor handles and analyzes the received sound signals. Once the abnormal sound information is found in these signals, the control instructions are sent to the controller and the voice player immediately, so as to control the driving of vehicles and warn the drivers. The audio signal processing and analysis is the core algorithm software of the device.

158

4 Intelligent Monitoring Technology of Automobile Power …

4.2.2 Intelligent Identification Algorithm of Abnormal Phenomena in Power and Transmission System How to identify and judge the abnormal phenomena of automobile power (engine) and transmission mechanism (system)? The commonly used methods are: (1) A mathematical model is established to accurately describe the actual operating conditions of the engine and the transmission mechanism, and many of experiments are carried out as a theoretical basis. The standard mode of the power system running state based on the power spectrum characteristics of the current signal is further given. Then the pattern recognition method based on the grey correlation degree calculation is used to identify the normal and fault status of the power system. (2) A feature-based fault information fusion identification method is proposed, that is, from the monitoring signal (parameters), through the signal feature extraction and information fusion to identify and judge the phenomenon characterization signal. It must be pointed out that the former (mathematical model method) needs to rely on the accurate mathematical model of the operation state of the power system to describe and establish the standard mode of the running state of the power system based on the power spectrum characteristics of the current signal. However, it is very difficult to establish an accurate mathematical model of the actual operating conditions of the power system. Therefore, the scope of its application has some limitations. The latter (based on the recognition method of feature information) is overly dependent on the sensing technology of parameter detection. So the redundancy of its information fusion is obviously narrowed. The intelligent recognition of the abnormal phenomena of the automobile power and transmission system is realized through the processing and analysis of the sound signals generated by the system to identify and judge the operation information of the system. The method of implementing this process is realized by support vector machines. Support Vector Machines (SVM) is a learning machine that is developed based on statistical learning theory aiming at solving the problem of finite sample machine learning. In other words, SVM is based on the VC dimension theory of statistical learning theory and the principle of structural risk minimization. It seeks the best tradeoff between the complexity of the model (learning accuracy for a specific training sample) and the learning ability (the ability to identify arbitrary samples without error) of the limited sample information to get the best promotion ability. SVM can effectively avoid many problems, such as over learning, under learning, “Curse of dimensionality”, and falling into local minimum in classical learning methods. SVM is developed from the optimal classification surface in the linear separable case. The strategy is to keep the empirical risk value fixed and to minimize the confidence range.

4.2 Intelligent Identification Technology of Power and Transmission …

159

Fig. 4.11 SVM architecture diagram

4.2.2.1

A Brief Introduction to SVM Theory

1. The architecture of SVM The construction idea of SVM can be summed up as follows: (1) In the process of constructing nonlinear mapping from input vector to highdimensional feature space, the feature space is hidden in the input and output. (2) The purpose of constructing an optimal hyperplane is to separate the features found in the first step. The architecture of SVM is shown in Fig. 4.11. In the graph, the left x1 , x2 , . . . , xN represents the different components of the N dimensional input vector x. K(x, xi ) represents the neuron transfer function that maps input data to M dimensional feature space. The output neurons of the right side use the hard limit transfer function, and the fixed offset b and the input data mapped to the feature space are used to realize the optimal hyperplane classification in the feature space. d represents the output value of SVM. 2. Optimal classification hyperplane The so-called optimal classification requires that the classification not only separates the two types of samples correctly (the training error rate is 0), but also makes the maximum classification interval. Consider the training sample {(xi , di )}Ni1 , where xi is the i example of the input pattern. di is the corresponding expected response output (target output). Assume that patterns represented by subsets of di  +1 and di  −1 are linearly separable, the decision surface equation used in the form of hyperplane separation is w·x+b0

(4.7)

160

4 Intelligent Monitoring Technology of Automobile Power …

In the formula, w is an adjustable weight vector, x is the input vector, b is the offset, and w  [w1 , w2 , . . . , wN ]  [wi ]; i  1, 2, . . . , N

(4.8)

x  [x1 , x2 , . . . , xN ]  [xi ] ; i  1, 2, . . . , N

(4.9)

T

T

The classification equation, that is, the decision surface Eq. (4.7) is normalized so that the linearly separable sample (xi , di )(i  1, 2, . . . , N , x ∈ RN , di ∈ {±1}) is satisfied di (w · x + b) ≥ 1; i  1, 2, . . . , N

(4.10)

For a given weight vector w and a bias b, the interval between the hyperplane defined by Eq. (4.7) and the nearest data point is called the separation edge. The distance between the positive and the negative sides of the hyperplane is called the classification interval, which is represented by ρ. The purpose of SVM is to find a special hyperplane, which makes the hyperplane interval ρ maximum. Under this condition, the decision surface is called Optimal Hyperplane. There are equations on both sides of the decision surface  w · x + b ≥ 0; if di  +1 (4.11) w · x + b < 0; if di  −1 Let w0 and b0 denote the optimal value of the weight vector and bias, then the optimal hyperplane of the decision surface is w0 · x + b0  0

(4.12)

g(x)  w · x + b

(4.13)

Defining discriminant function

An algebraic metric for distance from w to an optimal hyperplane is given, and x is expressed as x  xp + r

w w

(4.14)

Among them, xp is the regular projection of x on the optimal hyperplane, and r is the expected algebraic distance. If x is on the front of the optimal hyperplane, r is positive. Conversely, if x is negative in the optimal hyperplane, r is negative.

4.2 Intelligent Identification Technology of Power and Transmission …

(a)

(b)

x2

w 0 ⋅ x + b0 = 0

w

x2

w 0 ⋅ x + b0 = 0

xp

161

w

x r

xp x

x1

r x1

Fig. 4.12 Schematic diagram of optimal classification hyperplane geometric meaning

The geometric meaning of formula (4.12)–(4.14) is shown in Fig. 4.12. In the graph, w is the normal vector of the optimal hyper-plane. In (a), x is at the front of the optimal hyperplane, so r is positive. In (b), x is in the negative of optimal hyperplane,   so r is negative. g xp  0 is defined by definition, thus it can be deduced g(x)  w0 · x + b0  rw0 

(4.15)

Or r

g(x) w0 

(4.16)

Therefore, the target test problem is translated into: for the given test sample data set Γ  {(xi , di )}, it is necessary to find the optimal hyperplane parameters w0 and b0 , we can see that D must satisfy the conditions  w0 · x + b0 ≥ 1; if di  +1 (4.17) w0 · x + b0 < 1; if di  −1 When the formula (4.10) is established, it is shown that the model is linearly separable, and the formula (4.17) can be established by adjusting the values of w0 and b0 . If there is a data point (xi , di ), the equal sign of formula (4.17) is established, this point is called a support vector point, and SVM gets its name. Support vector is the set of data points closest to the decision surface. These data points are the most difficult to classify, so they are directly related to the optimal location of the decision surface. Consider a support vector xˆ corresponding to di  +1, according to the definition, there are   g xˆ  w0 · x ∓ b0  ∓1; in di  ∓1

(4.18)

162

4 Intelligent Monitoring Technology of Automobile Power …

r Classification interval

ρ

Fig. 4.13 Optimal classification diagram under linearly separable conditions

The algebraic distance from the support vector xˆ to the optimal hyperplane is ⎧   ⎨ w+1  ; if di  +1 g xˆ 0  r (4.19) w0  ⎩ −1 ; if di  −1 w0  Among them, positive sign means that xˆ is in the positive plane of the optimal hyperplane, the negative sign means that xˆ is in the negative plane of the optimal hyperplane. Let ρ be the optimal value of separating the edge distance between two classes, and two classes constitute the training set as Γ , so get ρ  2r 

2 w0 

(4.20)

It is shown in the upper form that the classification interval between the two classes is maximized equivalent to the Euclidean norm when the weight vector w is minimized. The optimal hyperplane defined by the formula (4.12) is unique, which means that the optimal weight vector w0 provides the maximum possible separation between the positive and negative cases. This optimization condition is obtained by Euclidean norm when the weight vector w is minimized. See Fig. 4.13, which is the illustration of the best classification idea with two dimensions as an example. The solid point and the hollow point represent two kinds of samples, H is the line of classification, H1 and H2 are the straight lines of the nearest sample H of each category, and parallel to the classification line H , The distance between H1 and H2 is called the classification interval ρ, and the distance from H1 (or H2 ) to the line H is r.

4.2 Intelligent Identification Technology of Power and Transmission …

163

3. The method of finding optimal hyperplane (1) Classification operations of linearly separable patterns To find the optimal hyperplane, the following two programming problems (minimization of functional) are needed φ(w) 

1 1 w2  wT w 2 2

(4.21)

The constraint conditions are as follows di (w · x + b) ≥ 1; i  1, 2, . . . , N The solution of this optimization problem is given by the saddle point of the following Lagrange function  1 T w w− αi [di (w · xi + b) − 1] 2 i1 N

L(w, b, α) 

(4.22)

Among them, αi is the Lagrange coefficient (also known as Lagrange multiplier), αi ≥ 0. At the saddle point, L takes the minimum, then w  w0 , b  b0 satisfies ⎧

N ⎪  ⎪ ∂L(w,b,α)

⎪  0 ⇒ αi di  0

⎪ ∂b ⎨ w  w0 , b  b0 , α  α0 i1 (4.23)

N ⎪  ⎪ ∂L(w,b,α)

⎪ ⎪ ∂w

 0⇒ αi di xi  w0 ⎩ ww ,bb ,αα 0

0

i1

0

That is the solution of the constrained optimal problem which is determined by the saddle point of the Lagrange function, and the Lagrange function must be minimized for w and b, and the α must be maximized. The saddle point corresponds to every Lagrange multiplier αi , the product of the multiplier and its corresponding constraint is 0, that is αi [di (w · xi + b) − 1]  0; i  1, 2, . . . , N

(4.24)

Only the multipliers satisfying exactly the upper form can assume nonzero values. When the optimal Lagrange multiplier is expressed by α0,i , the optimal weight vector w0 can be calculated w0 

N 

α0,i di xi

(4.25)

i1

The optimal bias b0 can be calculated using the obtained w0 . For a positive support vector, there is

164

4 Intelligent Monitoring Technology of Automobile Power …

b0  1 − w0 xˆ ; If di  +1

(4.26)

(2) Classification calculation of linear non-separable data points The optimal classification hyperplane is discussed on the premise of linearly separable. Most pattern recognition problems are in the original sample space. The sample points are linear non-separable. The so-called linear non-separable, that is, some training samples can not meet the formula (4.10) conditions. The above linear separable pattern classification algorithm is applied to linear non-separable data, and no feasible solution can be found, this problem can be verified by the arbitrary increase of the objective function. In the case of online non separable, SVM can use a nonlinear mapping function RN → F to map the original space sample to the high dimensional feature space F (possibly infinite dimension), and then construct the optimal classification plane in the high-dimensional feature space. For a set of training data, it is impossible to establish a separation hyperplane without classification error. It is still necessary to find an optimal hyperplane, it minimizes the probability of classifying the whole training set with large mean error. Definition: If the data point (xi , di ) does not satisfy the condition di (w · xi + b) ≥ +1; i  1, 2, . . . , N

(4.27)

There are two kinds of things that happen (1) The data point (xi , di ) falls within the classification interval, but on the right side of the decision surface. (2) The data point (xi , di ) falls within the classification interval, but on the wrong side of the decision surface. The separation edge between classes is soft at this time. For the case (1), the classification is still correct, and for the case (2) the classification is incorrect. In order to establish the method of processing non-separable data points, a set of nonnegative scalar variables {ξi }Ni1 is introduced into the definition of decision surface di (w · xi + b) ≥ 1 − ξi ; i  1, 2, . . . , N

(4.28)

In the form, ξi is called relaxation variable, which is used to measure the deviation degree of a data point to the ideal condition of the module separable. N  When the error is generated, the corresponding ξi must be consistent, so ξi i1

is an upper bound of training error number. For 0 ≤ ξi ≤ 1, it indicates that the data points fall into the interior of the classification interval, but on the right side of the decision surface. For ξi > 1, the data points fall to the wrong side of the classification hyperplane. Support vector is a set of special data points satisfying di (w · xi + b) ≥ 1 − ξi accurately.

4.2 Intelligent Identification Technology of Power and Transmission …

165

To find the separation hyperplane (decision surface) with the smallest average classification error on the training set, it is necessary to minimize the function of the weight vector w Φ(ξ ) 

N 

I (ξi − 1)

(4.29)

i1

The above functionals satisfy the constraints di (w · xi + b) ≥ 1 − ξi and the restrictions on w2 . I (ξ ) is an index function, define as follows  0; If ξ ≤ 0 (4.30) I (ξ )  1; If ξ > 0 The minimization of Φ(ξ ) for w is a nonconvex optimization problem, therefore, N  Φ(ξ )  ξi can be used to approximate functional. i1

At this point, the generalized optimal hyperplane can be further evolved to find the minimum of the function under the constraint of the condition (4.28). The function is N k  1 T Φ(w, ξ )  w w + C ξi 2 i1

(4.31)

In order to calculate conveniently, take k  1. Among them, the parameter C is the positive constant specified by the user, it controls the complexity of the learning machine and the balance between the non separable points, in other words, it actually plays a role in controlling the degree of punishment for the wrong samples, achieving the tradeoff between the proportion of misclassification samples and the complexity of the algorithm. Specifying a larger C can reduce the number of misclassification samples. At this point, the Lagrange function corresponding to the objective function is   1 T w w− αi [di (w · xi + b) − 1 + ξi ] − μi ξi 2 i1 i1 N

L(w, b, α, ξ ) 

N

(4.32)

Among them, the introduction of μi is to enhance the degree of certainty of ξi . Considering the KKT (Karush-Kuhn-Tucker) condition  αi [di (w · xi + b) − 1 + ξi ]  0 ; i  1, 2, . . . , N (4.33) (C − αi )ξi  0 Can get

166

4 Intelligent Monitoring Technology of Automobile Power …

 ∂L  0 ⇒ w0  αi di xi ∂w i1

(4.34)

 ∂L αi di 0⇒0 ∂b i1

(4.35)

N

N

∂L  0 ⇒ C − αi − μi  0 ∂ξi ξi ≥ 0, αi ≥ 0, μi ≥ 0

(4.37)

αi [di (w · xi + b) − 1]  0; i  1, 2, . . . , N

(4.38)

μi ξi  0

(4.39)

(4.36)

As mentioned earlier, we can use KKT to supplement the conditions, and determine the threshold by equality (4.32) and (4.34). The combination Eqs. (4.36) and (4.39) can be seen, if αi < C, get ξi  0. At the same time, the dual problem of the generalized optimal classification surface is almost the same as that of the linear separable case, in the case of linear separable, αi ≥ 0 becomes 0 ≤ αi ≤ C

(4.40)

In other words, we take all the data points satisfying 0 ≤ αi ≤ C in the training sample to participate in the calculation, it is better to obtain the average value of the obtained b0 as the final optimal bias. 4. A brief introduction to VC dimension theory The VC (Vapnik-Cervonenkis) dimension theory proposed by the early statistical learning theory provides an effective theoretical framework for measuring the complexity of forecasting models. However, the early VC dimension theory is only based on the empirical risk minimization principle, that is, the model with the least average training error as the final model of expectation. Therefore, the early statistical learning theory has been in the abstract theory and the concept of exploration, until the early 1990s, VC dimension theory has not been well applied. In the middle of the 1990s, Vapnik and his AT&T Bell laboratory group proposed a Support Vector Machines (SVM) algorithm based on VC dimension theory, that further enriched and developed the theory of statistical learning, so that it is not only a theoretical analysis tool, but also a prediction learning algorithm that can construct multidimensional prediction function, and can transform abstract learning theory into general practical prediction algorithm. SVM algorithm, which is developed based on statistical learning theory, is a special learning method for limited sample prediction. Compared with traditional statistics, the SVM algorithm is not based on the traditional empirical risk minimization principle, it is based on the principle of Structural Rask Minimization (SRM) and develops into a new structured learning method. It can be a good solution to high

4.2 Intelligent Identification Technology of Power and Transmission …

e

f

e

g e

e

e

f g

f

e

g

g

f g

f

167

e

f g

f g

e

f g

Fig. 4.14 3 Points scattered by a directed line in the two-dimensional plane

dimensional model construction problem of limited number of samples. Moreover, the constructed model has good prediction performance. As the basis of SVM algorithm, VC dimension theory and structural minimization principle also provide a theoretical basis and a unified theoretical framework for the further improvement of traditional statistical forecasting methods and empirical nonlinear prediction methods. (1) Boundary theory and VC dimensional principle The boundary theory mainly contains two parts: one is the non-constructive boundary theory, which can be obtained by the concept based on the growth function. The two is the tectonic boundary, it is the main problem by constructing the concept to estimate these functions. The latter includes the following contents: The VC dimension theory describe function can be given the number of categories, and uniformly convergent generalization boundary. According to whether the distribution of samples is independent, different learning functions can be obtained respectively, and the constructive boundary of learning machine generalization ability can also be controlled. The VC dimension describes function learning model collection capacity, in other words, the learning ability of the set of functions is described. VC dimension is bigger, function set is greater, its corresponding learning ability is strong. For example, for the two classification problem, n is the maximum number of points that can be divided into two categories by 2n methods using the machine function set, that is to say, for each possible partition, there is a function fα in the set of functions, so that the function takes 1 for one class and takes −1 for another class. See Fig. 4.14, take 3 points on the real two-dimensional plane R2 , the names of the three points are e, f and g respectively. The set of functions {f (x, a)} is a set of directed line sets. Obviously, the 3 points e, f , g can be divided into 23 categories (eg, f ), (ef , g), (gf , e), (egf , φ), (f , eg), (g, ef ), (e, gf ), (φ, efg) (see from left to right, from top to bottom in Fig. 4.14). Among them, the first indication of two tuple is +l class, second indication of two tuple is −1 class. φ is the empty set. For any partition, we can find a directed line in the function set corresponding to it. The direction of the directed line is +1 Class, and the reverse indicates −1 class. At this point, the VC dimension of functions set is equal to 3.

168

4 Intelligent Monitoring Technology of Automobile Power …

(2) Promotion capability boundary By controlling the promotion capability of learning machines, the inductive learning process suitable for small sample learning can be constructed. According to the statistical learning theory, for any w ∈ A (A is an abstract parameter set), the real risk R(w) is estimated, and the following inequality is satisfied by the probability of at least 1 − η   h (4.41) R(w) ≤ Remp (w) + ϕ N   Among them, Remp (w) is empirical risk, ϕ Nh is confidence risk, and      h ln 2N + 1 + ln η4 h h ϕ  N N

(4.42)

In the formula, N is the number of samples, the parameter h is the VC dimension of a set of functions, for linear classifier, satisfied h ≤ W2 R2 + 1

(4.43)

Among them, W is the training sample vector, R is the smallest ball radius of envelope training data. The basic idea of the structural risk minimization criterion is that the learning process not only minimizes the empirical risk, but also makes the VC dimension as small as possible, so that the future sample will have better generalization ability. (3) Inductive principle of structural risk minimization The so-called structural risk minimization (SRM) induction principle, i.e. Inequality (4.41) in R(w) and Remp (w) two common trade-offs, tends to a minimum in order to minimize the risk. At the same time, in the case that the empirical risk of the learning model is minimal, the generalization ability of the learning model can be large when the VC dimension h value is as small as possible (i.e., the confidence risk is as small as possible). According to the formula of risk estimation (4.41), when the number of training samples N is fixed, there are two parameters to control the risk R(w): empirical risk Remp (w) and VC dimension h. Among them, (a) The empirical risk Remp (w) depends on the function f (x, w) chosen by the learning machine, and the empirical risk can be controlled by controlling the w. (b) The VC dimension h depends on the set {f (x, w)} of functions that the machine is working on. To obtain the control to h, the function set can be structured, and the relationship between h and each function sub structure can be established. The purpose of controlling VC dimension h can be achieved by the choice of function structure.

4.2 Intelligent Identification Technology of Power and Transmission …

169

Fig. 4.15 Functional set diagram determined by nested subsets of functions

Sk S2

Let Sk  {f (x, w)|w ∈ Ak ; k  1, 2, . . . , ∞ }, and ⎧ ⎪ ⎨ S1 ⊂ S2 ⊂ · · · ⊂ Sk ⊂ · · ·   ⎪ ⎩ S  Sk

S1

(4.44)

k

This is the set of functions determined by nested sets of functions, see Fig. 4.15. 

After structuring {f (x, w)}, in structure S, any element Sk has a finite VC dimension hk , and h1 ≤ h2 ≤ · · · ≤ hk ≤ · · ·

(4.45)

The inductive principle of structural risk minimization,  for a given set of samples  (x1 , w1 ), (x2 , w2 ), . . ., (xN , wN ), a function f x, w(k) is selected in the subset Sk of functions to minimize the empirical risk Remp (w), and T also ensures that the confidence risk Sk (i.e., the VC dimension h) is minimal. The statistical law between the risk and the VC dimension shown by the inductive principle of structural risk minimization is shown in Fig. 4.16. The figure shows that the empirical risk Remp  (w) decreased with the increase of VC dimension h. Confidence will risk ϕ Nh increased gradually with the increase of VC dimension h. Moreover, the two parameters of VC dimension h can be used to determine the deficient learning and over learning regions of learning machine, that is, the R(·) region determined by h < h− is the lack of learning region, and the R(·) region determined by h > h+ is the over learning region. In the figure, the real risk of the boundary line is real curve R(w).

4.2.2.2

Systematic Learning

The so-called learning use the intelligent recognition device for the abnormal phenomena of power and transmission system to learn the sound signals of the automobile starting and driving system under the normal working condition, attempting to establish the characteristic space of the automobile launching and driving system

170

4 Intelligent Monitoring Technology of Automobile Power … Deficient learning

Over learning

R ( w) Risk

⎛h⎞ ⎟ ⎝N⎠

ϕ⎜

Confidence risk

The bounds of real risk

Confidence range Empirical risk

h−

hk

h+

Remp ( w )

h

Fig. 4.16 Schematic diagram of inductive principle of structural risk minimization

under normal sound condition. The process of sampling, characteristic transformation and establishing characteristic space of the sound signals of automobile starting and transmission system is called the learning process of sound information of automobile starting and transmission system. The learning process is carried out on the automobile test bench, and the detailed steps are as follows. 1. Audio signal acquisition The sound signals are collected from the engine and the transmission mechanism respectively by the first and second pickup. These sound signals include sound signals from car start, speed up, different engine speeds, different vehicle speeds, vehicle deceleration and braking conditions. Pickup sound signal on behalf of different working conditions were collected by sampling period set in advance and sent to the signal processor. 2. Analog to digital conversion In the signal processor, the analog signal of sound is converted into digital signal and processed by the arithmetic module. 3. Fourier transform of time varying signals The Fourier transform is first performed on the sound digital signal x(t)  ∞ X (jω)  x(t)e−jωt dt

(4.46)

−∞

In the formula, ω  2π f , whose unit is radian/ second, and when X (jω) is expressed as form |X (jω)|ejϕ(ω) , the curves of |X (jω)| and ϕ(ω) changing with ω can be obtained, which are called amplitude frequency characteristics and phase frequency characteristics of x(t). The frequency distribution information in the sound signal can be obtained by the formula (4.46).

4.2 Intelligent Identification Technology of Power and Transmission …

171

4. Structure two types of training samples for automotive engines (1) The training sample vector is set up for the acoustic signals emitted by automobile engines under various working conditions T  x1  x11 x21 . . . xN1 −1 xN1 T  x2  x12 x22 . . . xN2 −1 xN2

(4.47) (4.48)

Among them, the x1 and x2 are separately the training sample vectors of the amplitude frequency characteristic vectors (1 types) of the automobile engine under normal conditions and the amplitude frequency characteristic vector (2 types) of the automobile engine under abnormal conditions. Each training sample in the two training samples corresponds to the sampling point information of the working condition. According to these samples, the matrix of training samples and the matrix of corresponding class samples can be constructed. The numerical value in the class sample matrix is the classification category of the sample, because there are only two types: audio information under normal working conditions and audio information under abnormal conditions, the value of the sample class matrix is 1 or −1. The training sample matrix consists of 1 categories and 2 categories of training samples is X  x1 x2 , and the corresponding class sample matrix is T  1 1 ... 1 d . −1 −1 . . . −1 (2) Solving the Lagrange multiplier αi Using the formula L(w, b, α)  21 wT w −

N 

αi [di (w · xi + b) − 1], according to the

i1

SVM algorithm and the radial basis function in the kernel function   x − xi 2 K(x, xi )  exp − σ2

(4.49)

Calculate. In the formula, σ is the mean square-error. Here, each base function center corresponds to a support vector, and the support vector and the output weights are determined automatically by the algorithm. Therefore, it is possible to solve the Lagrange multiplier αi , i  1, 2, . . . , N corresponding to each training sample, and most of them are zero. Only a few αi which are not zero corresponds to the support vector.

172

4 Intelligent Monitoring Technology of Automobile Power …

(3) Obtaining bias b0 Using Lagrange multiplier αi , training sample matrix and corresponding class sample matrix, the following formula is adopted αi [di (w · xi + b) − 1] = 0 (i  1, 2, . . . , N ) Getting bias b0 . In the linear separable case, then w0 can be obtained according to N  the formula w0  α0,i di xi . In this way, the specific expression of the classification i1

decision function of the automobile engine running state can be obtained 5. Structure two types of training samples for automobile transmission (1) The training sample vector is set up for the sound signals emitted by automobile transmission under various working conditions T  y1  y11 y21 . . . yN1 −1 yN1 T  y2  y12 y22 . . . yN2 −1 yN2

(4.50) (4.51)

Among them, the y1 and y2 are separately the training sample vectors of the amplitude frequency characteristic vectors (1 types) of the automobile engine under normal conditions and the amplitude frequency characteristic vector (2 types) of the automobile engine under abnormal conditions. Each training sample in the two training samples corresponds to the sampling point information of the working condition. According to these samples, the matrix of training samples and the matrix of corresponding class samples can be constructed. The numerical value in the class sample matrix is the classification category of the sample, because there are only two types: audio information under normal working conditions and audio information under abnormal conditions, the value of the sample class matrix is 1 or −1. The training sample matrix consists of 1 categories and 2 categories of training samples is Y  y1 y2 , and the corresponding class sample matrix is T  1 1 ... 1 d . −1 −1 . . . −1 (2) Solving the Lagrange multiplier αi Using the formula L(w, b, α)  21 wT w −

N 

  αi di (w · yi + b) − 1 , according to the

i1

SVM algorithm and the radial basis function in the kernel function   y − yi 2 K(y, yi )  exp − σ2

(4.52)

4.2 Intelligent Identification Technology of Power and Transmission …

173

Calculate. In the formula, σ is the mean square-error. Here, each base function center corresponds to a support vector, and the support vector and the output weights are determined automatically by the algorithm. Therefore, it is possible to solve the Lagrange multiplier αi , i  1, 2, . . . , N corresponding to each training sample, and most of them are zero. Only a few αi which are not zero corresponds to the support vector. (3) Obtaining bias b0 Using Lagrange multiplier αi , training sample matrix and corresponding class sample matrix, the following formula is adopted   αi di (w · yi + b) − 1  0 (i  1, 2, . . . , N ) Getting bias b0 . In the linear separable case, then w0 can be obtained according to N  the formula w0  α0,i di yi . In this way, the specific expression of the classification i1

decision function of the automobile transmission mechanism running state can be obtained.

4.2.2.3

Online Identification Process

1. Collect audio signals from power and transmission system online and in real time The sound signal generated by the first, second pickup online real-time acquisition of the engine and the transmission, sent to the signal processor. 2. Analog to digital conversion In the signal processor, the analog signal of sound is converted into digital signal, and then it is handed over to the arithmetic module. 3. Fourier transformation

∞ Real time audio digital signal x(t) is transformed by X (jω)  −∞ x(t)e−jωt dt to Fourier transform, thus, the amplitude frequency characteristics and phase frequency characteristics can be obtained under different operating conditions. 4. Class determination of test samples

  2 i and K(y, yi )  Using classification decision function K(x, xi )  exp − x−x σ2   2 i to get class results in real time. exp − y−y σ2 (1) If w0 · x + b0 ≥ 1, it means that the current sample belongs to the first category, that is, the engine is in normal operation. On the contrary, this sample belongs to the second category, that is, there is an abnormal sound in the engine.

174

4 Intelligent Monitoring Technology of Automobile Power …

(2) If w0 · y + b0 ≥ 1, it means that the current sample belongs to the first category, that is, the transmission mechanism is in normal operation. On the contrary, this sample belongs to the second category, that is, the transmission mechanism appears abnormal sound. 5. Decision output The control instructions are output by the signal processor according to the classification results of the test samples. When the identification results confirm that the engine has abnormal sound, the device will automatically remind the driver through the voice player: “the engine has abnormal sound, need to stop the inspection!”. At the same time, the system will control the vehicle through the controller, such as deceleration or brake. Similarly, when the identification results confirm that the car transmission mechanism has abnormal sound, the device system will automatically through the voice player prompts the driver: “car transmission has abnormal sound, need to stop inspection!”. At the same time, the system will control the vehicle through the controller, such as deceleration or brake.

Chapter 5

Intelligent Vehicle Navigation and Traffic System

In the past ten years, the problem of urban traffic congestion has become a common concern of governments at all levels and traffic experts. In this chapter, an intelligent vehicle navigation and traffic system which can effectively alleviate urban traffic congestion is described. Among them, intelligent vehicle navigation is the application of wireless sensor network technology in automobiles. Intelligent vehicle navigation (or known as wireless sensor technology) is an integral part of the intelligent transportation system.

5.1 Summary Intelligent transportation system (ITS), it is a large system that integrates advanced information engineering, data communication, electronic control and computer technology into the whole traffic management system and vehicles. The system can achieve the coordination and control between traffic and vehicles in an all-round, real-time, accurate and efficient way. It can make the efficiency of the whole transportation system reach the highest scientific level. Intelligent transportation system is generally composed of advanced traffic management system, public transport operation system, advanced vehicle control system, emergency management system and advanced traffic information service system. In the intelligent transportation system, the vehicle as the main body of transportation is a very important part of the system. Automobile, as the main body of transportation, is the main regulating target of ITS. The functions of ITS must be supported by the corresponding technical equipment in the car. Such as: the dynamic intelligent navigation technology, communication technology and other information devices on the car.

© Shanghai Jiao Tong University Press, Shanghai and Springer Nature Singapore Pte Ltd. 2019 X. Zhang and M. M. Khan, Principles of Intelligent Automobiles, https://doi.org/10.1007/978-981-13-2484-0_5

175

176

5 Intelligent Vehicle Navigation and Traffic System

Automobile is one of the most important elements in intelligent transportation system. It has long been the focus of technological improvement. When the sensor is installed in a car, the car can be able to perceive all kinds of information on the way. Through wireless communication devices, a large number of sensor nodes on the vehicle can interact with the communication endpoints on the road infrastructure, which constitute the vehicle wireless sensor network. It is called VSN (Vehicular Sensor Networks). The vehicle wireless sensor network (WSN) builds the technical basis of the intelligent vehicle navigation system. The development and application of intelligent navigation technology has started since the 80 s of last century, so far it has more than 30 years of technology development history. Intelligent navigation integrates the technologies of automobile, traffic, computer, geographic information, communication, system science and so on. It has been the focus of many high-tech companies and universities. At present, it has played an important role in the transportation system of various developed countries. Vehicle intelligent navigation system is a terminal device in ITS, it is mainly based on the map of its storage and update information to receive real-time data, combined with the GPS (or the Beidou satellite navigation system) and gyroscope (or electronic compass) inertial navigation equipment, can provide drivers the vehicle location, the best route and location query, correct line yaw etc. In recent years, with the rapid development of communication technology, there are some new functions of vehicle intelligent navigation technology, that is, dynamic intelligent navigation based on real-time traffic information. The system can use the on-board communication equipment to receive real-time traffic information sent by the traffic information center, and combined with electronic map processing, so as to guide the driver to avoid traffic congestion, in real time, the driver chooses the best route to ensure that the driver can arrive at the destination in the quickest way. To ensure that the urban road network vehicles can be punctual, fast, safe and orderly operation, to achieve full and balanced use of urban road network capacity, improve traffic efficiency. For example, through communication between vehicles, we can realize accident alarm and hidden hints for drivers, help drivers avoid collision with other vehicles, and guide and dredge vehicles at key locations such as intersection and highway entrance. When driving on the highway, people can understand the situation of the road through the wireless communication system, and adjust the driving route according to the accident or traffic jam. Weather information, gas stations and restaurant locations can also be shared among vehicles. Traffic system can also adjust the traffic light time according to the traffic flow and traffic information. The difference between the vehicle sensor network and the common sensor network is that the energy and volume arrangement of the vehicle sensor nodes is not limited to the common sensor network. In cars, we can easily install strong processing units, wireless communication equipment, GPS, chemical sensors, cameras, vibration/sound sensors, etc. From the technical point of view, the intelligent vehicle navigation should include the two technologies (i.e.) positioning and intelligent selection of driving routes.

5.2 Automobile Positioning Technology

177

Fig. 5.1 Constellation diagram of GPS satellite

5.2 Automobile Positioning Technology 5.2.1 GPS Positioning GPS is the abbreviation of global positioning system. GPS was developed by the US Department of defense for military timing, positioning and navigation purposes.

5.2.1.1

Geometric Principle

GPS consists 24 communications satellites. Among them, there are 3 backup satellites, these satellites are distributed over 20,200 km above the earth’s surface and are divided into 6 orbital planes (see Fig. 5.1). The satellite orbit angle is 55°, providing global all-weather, once per second, continuous positioning signal. These satellites circle the earth every 11 h and 58 min, just like the moon, orbiting around the earth. These satellites require ground control stations to be monitored at any time to ensure that the GPS satellite operates properly on its correct orbit. At the same time, the ground monitoring center can upload data to the satellite, and the satellite can then transmit the information to the GPS terminal users. There are five monitoring centers on the ground, including four uploading data stations and a control center. These monitoring centers locate the satellite controlled by latitude. Conceptually, the GPS represents the entire system, including the sky satellite, ground control station and GPS receiver. In general, the GPS receiver receives only 4 or more satellites at any given time. The basic principle of GPS positioning is to measure the distance between the known satellite and the user receiver, and then integrate the data of multiple satellites to know the specific location of the receiver. The 2D positioning (longitude and latitude) can be done by three satellites, while the 3D positioning (longitude, latitude and height) can be done by four or more satellites. When the receiver continues to update its position, the direction and velocity of its movement can be calculated.

178

5 Intelligent Vehicle Navigation and Traffic System

First satellite position

Third satellite position Second satellite position Fig. 5.2 Geometric principle diagram of 2D positioning

The geometric principle of 2D positioning is shown in Fig. 5.2. It can be seen from the figure, while receiving two transmitted source signals may have two points A and B, A and B from the first signal transmitted source (the first satellite) and second signal transmitted source (second satellites) radius is R1 and R2. If only two signals are received, it is impossible to determine whether they are in A or at B, therefore, it is necessary to use third signal sources to further confirm their location, as the A point in the graph.

5.2.1.2

The Composition and Structure of the GPS Utility System

A practical GPS positioning (navigation) system consists of three parts: space, ground control and user, as shown in Fig. 5.3. (1) Space part: the GPS satellite constellation consists of 21 working satellites and 3 spare satellites on orbit (see Fig. 5.1). (2) The ground control part includes: main control station, monitoring station and data injection station. (a) The main control station is located at Falcon Air Force Base in Colorado, USA. According to the observation data of the monitoring stations on GPS, the correction parameters of the satellite’s ephemeris and the satellite clock are calculated, and the data are injected into the satellite through the data injection station. The satellite is controlled and the instructions are issued to the satellite. When the working satellite has a technical fault, it will dispatch spare satellites to replace the failed working satellites. The main control station also has the function of the monitoring station at the same time. (b) The monitoring station is located at 4 stations of Hawaii, Ascencion, Diego, Garcia and Kwajalein, which is used to receive satellite signals and monitor the working state of the satellite.

5.2 Automobile Positioning Technology

179

Fig. 5.3 Three parts of the GPS utility system

(c) The data injection station is located at Ascencion, Diego, Garcia and Kwajalein. The role and function of the master station is calculation of satellite ephemeris and satellite clock correction parameters into the satellite. (3) The user part is composed of the GPS (or the Beidou satellite navigation system) receiver, the meteorological instrument, the computer and so on. GPS (or Beidou satellite navigation system) should achieve a goal in the process of specific use. That is to say, the position of the satellite can be detected in the satellite ephemeris according to the time recorded by the satellite clock. The distance from the user to the satellite is obtained from the satellite signal to the user’s time multiplied by the speed of light.

5.2.1.3

Navigation Message

When the GPS satellite works normally, it will continuously use the pseudo-random code composed of binary codes to launch the navigation message. There are two kinds of pseudo code used in GPS system, which are civilian C/A code and military P (Y) code. The frequency of C/A code is 1.023 MHz, the repetition period is 1 ms, the code spacing is 1 µs, which is equivalent to 300 m. The frequency of P code is 10.23 MHz, the repetition period is 266.4 days, the code spacing is 0.1 µs, which is equivalent to 30 m. The Y code is formed on the basis of P code, and its secrecy performance is stronger. Navigation message includes satellite ephemeris, working condition, clock correction, ionospheric time delay correction, atmospheric refraction correction and other information. It is demodulated from the satellite signal and transmitted by 50 b/s

180

5 Intelligent Vehicle Navigation and Traffic System

modulation on carrier frequency. Each frame of the navigation message contains 5 sub frames, each of which has a length of 6 s. The first three frames each having 10 letters, repeat every 30 s, update once every hour. In the latter two frames, there are 15,000 b. The main contents of navigation message are telemetry code, conversion code and first, second, third data block, and the most important one is ephemeris data. When the user receives the navigation message, the satellite time is extracted and then compared with their own clock, the distance between user and satellite is extracted, and then calculate the satellite launch message location using satellite ephemeris data in navigation message. Thus, the location, speed and other information of the user in the WGS-84 geodetic coordinate system can be known. Therefore, the role of GPS navigation satellite system is to continuously transmit navigation messages. However, the clock used by the user receiver can not always be synchronized with the satellite on-board clock. Therefore, in addition to the 3D coordinates of the user, the time difference between a satellite and the receiver is also introduced as an unknown number, and then the 4 unknowns are solved by 4 equations. This is because if you want to know the coordinates of your receiver, you should at least receive the signals from the 4 satellites. That is to say, the GPS (or the Beidou satellite navigation system) receiver can receive a time information that can be used to give accurate time to nanosecond, a forecast ephemeris used to predict the general position of the satellite within the next few months, there is also a broadcast ephemeris for calculating the satellite coordinates needed for positioning, its accuracy reaches several meters to tens of meters (all satellites are different and will change at any time), and will receive GPS system information, such as satellite status, etc. GPS (or Beidou satellite navigation system) receiver can get the distance from the satellite to the receiver by measuring a “code”. Due to the error of receiver clock and ionospheric interference, the distance is actually not the true distance between the user and the satellite, so it is called pseudo range (PR). The pseudo distance measured by 0A code is called the pseudo range of UA code. The precision is about 20 m. The pseudo distance measured by P code is called the pseudo distance of the P code, and the precision is about 2 m. The receiver of GPS (or Beidou satellite navigation system) needs to decode the received satellite signals. Strictly speaking, carrier phase should be referred to as carrier beat phase. It receives the difference between the carrier phase of the satellite signal affected by the Doppler frequency shift and the signal phase generated by the local oscillation of the receiver. In general, as long as the satellite signal is tracked, we can record the change of the phase when measuring the calendar time determined by the receiver clock. The initial value of the observation of the receiver and the phase initial value of the satellite oscillator are not known, and the phase integer of the starting calendar is not known, so only the phase observation value can be used only when it is relative positioning and having a continuous observation value, and the position accuracy that is better than the meter level can only be used in phase observation.

5.2 Automobile Positioning Technology

181

Fig. 5.4 Schematic diagram of automobile GPS positioning

5.2.1.4

Positioning Mode

According to the positioning mode, GPS positioning is divided into single point positioning and relative positioning (differential positioning). Single point positioning is based on the observation data of a receiver to determine the location of the receiver, it can only use pseudo distance observation, which can be used for the general navigation and positioning of vehicles and ships. Relative positioning (differential positioning) is a method to determine the relative position between observation points based on the observation data of more than two receivers, it can be used either pseudo range observation or phase view measurement, phase observation should be used for relative positioning in geodetic surveying or engineering surveying. In the observation parameters of GPS, there will be errors of the clock difference between the satellite and the receiver, the delay of atmospheric propagation, the multipath effect and so on, which is inevitable. In the location calculation, these parameters are also influenced by the satellite ephemeris error. In the relative position, most of the common errors will be offset or weakened, so the positioning accuracy will be greatly improved. In this case, a dual frequency receiver can counteract the main part of the ionosphere error in the atmosphere according to the observation of two frequencies. If the precision is high and the distance of the receiver is far away (there is a significant difference in the atmosphere), it is better to choose a dual frequency receiver. The positioning process of the vehicle GPS receiver using three satellites is shown in Fig. 5.4.

5.2.1.5

Error Sources in GPS Positioning

(1) Satellite related errors (a) The error of satellite ephemeris is caused by the difference between the actual position of the satellite and the position of the satellite given by the satellite ephemeris.

182

5 Intelligent Vehicle Navigation and Traffic System

Fig. 5.5 Ionosphere

The size of ephemeris error mainly depends on the quality of satellite orbit determination system, the number and accuracy of observation, the mathematical mechanics model and the perfection of orbit determination software. In addition, there is a direct relationship between the extrapolation time interval of ephemeris (the extrapolation time interval of observed ephemeris can be regarded as zero). The effect of track error δr on baseline error δb can be expressed in the lower form δb ≈

D δr ρ

(5.1)

In the formula, D is the baseline and ρ is the distance from the satellite to the earth, p ≈ 25,000 km. (b) The satellite clock error, which contains systematic errors such as clock error, clock speed, frequency drift, etc., also contains random errors. Although high precision atomic clocks have been used on satellites, their errors are unavoidable. (2) Errors related to signal propagatio (a) Ionospheric delay, the ionosphere (including stratosphere) is in the atmosphere between 50 and 1000 km (see Fig. 5.5). The existence of charged particles in the ionosphere will affect the propagation of radio signals, the charged particle will change the propagation speed of the radio signal, and the propagation path is bent. Thus, the product of the signal propagation time t and the speed of light c in the vacuum is not equal to the geometric distance between the satellite and the receiver. The ionospheric delay depends on the total electron content TEC and the frequency f of the signal on the propagation path. TEC is related to many factors, such as time, location, sunspot number and so on.

5.2 Automobile Positioning Technology

183

(b) Tropospheric delay, the troposphere is below 50 km altitude of the atmosphere. The propagation speed of GPS satellite signal in troposphere is V  ηc . Among them, c is the speed of light in vacuum, η is the atmospheric refractive index, and its value depends on the temperature, air pressure and relative humidity and other factors. (c) Multiple path errors, a signal that arrives at the receiver after being reflected on the surface of some object, if the signal is superimposed to the receiver directly from the satellite, it will cause the system error to be measured. The influence of multiple path errors on ranging code pseudo range observation is much greater than that of carrier phase observation. Multiple path errors depends on the environment around the station, the performance of the receiver and the length of the observation time. (3) The error related to the receiver (a) The receiver clock error mainly depend on the quality of the clock, and it also has a certain relationship with the application environment. It has the same effect on ranging code i.e. pseudo range observation and carrier phase observation. (b) The position error of the receiver is due to the fixed location of the receiver. When it’s time service and orbit determination, the inherent error of the receiver will cause the system error of the timing and orbit determination results. (c) Measurement noise of receiver, when the receiver is used for GPS measurement, the instrument and its external environment influence can cause random measurement error, Its value depends on the performance of the instrument and the working environment. The effect of measurement noise is usually negligible when the observation is long enough. (4) Relativistic effect The relative clock error between the satellite clock and the receiver clock caused by the different motion states and gravity positions of the satellite clock and the receiver clock is the phenomenon of relativistic effect. (a) Special relativity If the motion speed of the satellite in the geocentric inertial coordinate system is VS , then the clock on the ground frequency f will be placed on the satellite, and its frequency f S will become  fS  f 1 −



VS c

2 0.5

  VS2 ≈ f 1− 2 2c

(5.2)

The frequency difference between the two is  f1  f S − f  −

VS2 · fS 2c2

(5.3)

184

5 Intelligent Vehicle Navigation and Traffic System

(b) General relativity According to the general relativity, if the gravity position of the satellite is W S , the gravity position of the earth station is WT , then the same clock in the satellite or ground the frequency difference is as follows   1 W S − WT μ 1 − (5.4) · f  2· f ·  f2  c2 c R r where, R and r are the radius of the earth and the rotation of the satellite around the earth respectively, μ  3.986005 × 1014 m3 s2 (C) Total influence  f   f 1 +  f 2  4.449 × 10−10 f

(5.5)

General relativistic effects of a clock on satellite will increase the frequency of 4.449 × 10−10 f than on the ground, therefore the simplest way to solve the relativistic effect is to reduce the clock frequency by 4.449 × 10−10 f in the manufacture of the satellite clock. (5) Other factors (a) The GPS control is affected by human or computer. (b) The error caused by the control part of GPS or the error introduced by users during data processing. (c) The imperfection of the data processing software algorithm will have an impact on the satellite positioning results.

5.2.2 Car GPS and GIS The vast majority of individual GPS receivers use longitude, latitude and altitude to indicate their position. In practical application, GPS vehicle location should also be combined with geographic information system (GIS) to have practical function. The geographic information system (GIS) contains a map database stored in computer memory. GIS is more sophisticated than a paper map. In addition to displaying three-dimensional topographic maps, it can also include other useful information, such as gas stations, tourist attractions, and so on. As shown in Fig. 5.6, it is the GIS example of a city’s information. The vehicle navigation system is actually a combination of GPS (or Beidou satellite navigation system) and GIS. The GIS (or the Beidou satellite navigation system) used for automobile navigation is mainly to extract the digital electronic map part of the GIS database. The fundamental difference between an electronic map and a common map is that the former is hierarchical, and the latter is not hierarchical. For example, electronic maps can be divided into road map, site electronic map, unit electronic map and so on. The road electronic map can be decomposed into: highway electronic map, National

5.2 Automobile Positioning Technology

185

Fig. 5.6 GIS legend of city information. a Urban geography map, b City street ma

Road electronic map, and various levels (such as one or two, three level) of highway electronic map and so on. See Fig. 5.7. Users can add or delete layers according to their purposes. We can also select different electronic map layers to arrange and produce different display effects. Electronic map is the basis of GPS positioning and navigation, which is composed of longitude, latitude, road, ground features and landforms, so on. In navigation, the latitude and longitude data received by GPS can be displayed on the corresponding level, through the road data layer covering it, we can know the location of the vehicle. With the movement of the vehicle, the marking of the location of the vehicle will move along with the geographical location determined by GPS. As shown in Fig. 5.7, the images of the 4 levels show the latitude and longitude, the administrative divisions and the urban roads of Shanghai and its surrounding cities respectively from top to bottom.

5.3 Intelligent Traffic Route Selection Technology The intelligent selection of driving routes is commonly known as the “Vehicle navigation”. Vehicle navigation technology based on GPS and GIS has been widely adopted. Figure 5.8 shows the man-machine interface of the vehicle navigation system. The practical value of vehicle navigation system is that it enables drivers to know their geographical location and the route needed to get to the destination at any time and anywhere. Route selection algorithm belongs to the category of optimal control, and involves a dynamic programming problem. This is a process of information fusion and decision making based on the multi relational data structure, as shown in Fig. 5.9.

186

5 Intelligent Vehicle Navigation and Traffic System

Fig. 5.7 Schematic diagram of hierarchical structure of electronic map

Fig. 5.8 Schematic diagram of vehicle navigation system

5.3.1 Basic Principles of Traffic Route Planning Generally speaking, the vehicle routing algorithm used in the vehicle navigation system is one of the dynamic programming (DP). The meaning of “Programming” in mathematics is “optimization”. Its basic principle is that whatever the initial state and

5.3 Intelligent Traffic Route Selection Technology

187

Unit map database Travel destination query Traffic flow database

Map information Station information Unit information

Road information fusion and decision making

Information processing Digital map database

Traffic flow information

Reminder and display of voice, data and map

Station map database

Fig. 5.9 Multi information fusion and decision flow chart

A

B1

C1

B2

C2

B3

C3

D

Fig. 5.10 Topological legend of road network

initial decision is, for all the states formed before decision, all decisions after it should constitute the best strategy. Therefore, it is necessary to make a brief introduction to the dynamic planning of the route.

5.3.1.1

Basic Concepts of Route Dynamic Programming

To illustrate the convenience, take the road network as shown in Fig. 5.10 as an example. In the figure, the number of paths represents the mileage, which aims to find the shortest route from A to D. Definition (1) Using k to represent stage variable, k  3 (2) The state variable sk has no aftereffect (that is, the future is only related to the current), and the state variable set is defined as Sk , such as S2  {B1 , B2 , B3 }

188

5 Intelligent Vehicle Navigation and Traffic System

(3) The u k (sk ) is a decision variable, u k (sk ) ∈ Uk (sk ), and the Uk (sk ) is the permission set, such as U2 (B1 )  {C1 , C2 , C3 }, the operation strategy is p1,n (u 1 (s1 ), u 2 (s2 ), . . . , u n (sn )) (4) sk+1  Tk (sk , u k ) is a state transition equation. (5) Function definition (a) The function of the index is as follows: dk (sk , u k ) is a stage index function (b) The process index function is as follows: The process from 1 to n is called the original process of the problem. The process from k to n is called the back sub-process of the original process.   The V1,n s1 , p1,n  represents the index function of the original process. The Vk,n sk , pk,n represents the index function of the back subprocess. The f k (sk ) represents the optimal index function of the rear, and     ∗ f k (sk )  Vk,n sk , pk,n  opt sk , pk,n (5.6) pk,n ∈P1,n

∗ In the formula, P1,n is a set of policies, pk,n is the value when pk,n obtains the optimal index function.

5.3.1.2

Route Dynamic Programming Process

The policy decision is divided into 3 sections (see Fig. 5.10). 1. CD segment Right now, k  3 ∵ f 3 (C1 )  d(s2 , u 2 )  d(C1 , D)  3, ∴ u ∗3 (C1 )  D ∵ f 3 (C2 )  d(s2 , u 2 )  d(C2 , D)  5, ∴ u ∗3 (C2 )  D ∵ f 3 (C3 )  d(s2 , u 2 )  d(C3 , D)  3, ∴ u ∗3 (C3 )  D 2. BC segment Right now, k  2 ∵ f 2 (B1 )  min {d(B1 , Ci ), f 3 (Ci )}  min{7 + 3, 6 + 5, 5 + 3}  8, i1,2,3

∴ u ∗2 (B1 )  C3 ∵ f 2 (B2 )  min {d(B2 , Ci ), f 3 (Ci )}  min{6 + 3, 5 + 5, 7 + 3}  9, i1,2,3

u ∗2 (B2 )

∴  C1 ∵ f 2 (B3 )  min {d(B3 , Ci ), f 3 (Ci )}  min{5 + 3, 6 + 5, 7 + 3}  8, i1,2,3

∴ u ∗2 (B3 )  C1

5.3 Intelligent Traffic Route Selection Technology

189

3. AB segment Right now, k  1 ∵ f 1 (A)  min {d(A, Bi ), f 2 (Bi )}  min{4 + 8, 4 + 9, 5 + 8}  12, i1,2,3

∴

u ∗1 (A)

 B1

So, the optimal strategy for the whole process is u ∗1 (A)  B1 , u ∗2 (B1 )  C3 , u ∗3 (C3 )  D That is, the optimal route is A → B1 → C3 → D The recurrence formula of route dynamic programming is as follows f k (sk )  min {d(sk , u k ), f k+1 (sk+1 )} uk

(5.7)

5.3.2 Intelligent Route Selection Algorithm The actual process of intelligent selection of traffic route is much more complicated than the basic principle of dynamic programming. The reasons are as follows: (1) The network topology is more complex. The state variable for the same stage variable is not the same. That is, the after effect of the different nodes is inconsistent. (2) The path of the front and back nodes in the network topology should be weighted. That is to say, under a specific optimization goal, the attributes of each path are inconsistent. For example, their fastest driving time is different for the same mileage highway and national road. (3) The optimization of route choice is often multi-objective process optimization. In other words, due to the different road grade and traffic flow, the fastest travel time is obviously different for the same mileage route.

5.3.2.1

Selection Method When Considering Road Grade

Definition α is defined as the weight of the road level. α ∈ [0.1, 1.0].The weight of the high grade road is greater than the weight of the low grade road. β is defined as the impact factor of the road level. β ∈ [1, 10], β  α1 . The influence factor of the high grade road is less than the influence factor of the low grade road. When the optimal target of the route selection is the fastest travel time, the mileage for each possible path must be removed by the weight α, or multiplied by the influence factor β, the equivalent mileage obtained by this method can represent its role in optimal selection.

190

5 Intelligent Vehicle Navigation and Traffic System

Again take the road network shown in Fig. 5.10 as an example. It is assumed that the route from B1 to C1 is the highway section, and the other roads belong to the first grade road. It is also assumed that the weight of the expressway is α0  1.0, and the weight of the first grade road is α0  0.6, then the calculation procedure mentioned above should be as follows: 1. The calculation of CD segment is invariant. 2. BC segment Right now, k  2

∵ f 2 (B1 )  min {d(B1 , Ci ), f 3 (Ci )}  min 7 + u ∗2 (B1 )

i1,2,3

3 6+5 5+3 , , 0.6 0.6 0.6



 12

∴  C1

∵ f 2 (B2 )  min {d(B2 , Ci ), f 3 (Ci )}  min 6+3 , 5+5 , 7+3  15 0.6 0.6 0.6 i1,2,3

∴ u ∗2 (B2 )  C1

∵ f 2 (B3 )  min {d(B3 , Ci ), f 3 (Ci )}  min 5+3 , 6+5 , 7+3 ≈ 13.3 0.6 0.6 0.6 i1,2,3

∴ u ∗2 (B3 )  C1

3. AB segment Right now, k  1 ∵ f 1 (A)  min {d(A, Bi ), f 2 (Bi )}  min i1,2,3

4+8 0.6

, 4+9 , 5+8  20 0.6 0.6

∴ u ∗1 (A)  B1

So, the optimal strategy for the whole process is u ∗1 (A)  B1 , u ∗2 (B1 )  C1 , u ∗3 (C1 )  D That is, the optimal route is A → B1 → C1 → D

5.3.2.2

Route Selection Example

Taking a vehicle from Jiangsu, Wujiang to Shanghai, Jinshan District as an example. 1. Displays the initial position Show the current vehicle position in the electronic map, as shown in Fig. 5.11: “Wujiang city”. 2. Select the destination and determine the route Click where you want to go to your destination, such as Shanghai, Jinshan district (see Fig. 5.12). The intelligent route selection algorithm can find the best route from the crisscross road network. As shown in Fig. 5.12a, it is an optional route to travel from Wujiang to Jinshan via Jiaxing. As shown in Fig. 5.12b, it is also an optional

5.3 Intelligent Traffic Route Selection Technology

191

Fig. 5.11 Sketch map of the current position of the vehicle

route to travel from Wujiang to Jinshan via Qingpu. It shows that the two routes have the optimal effect of close phase. Regardless of the one of the two, their algorithms are determined by the high weight of the freeway as a principle. Therefore, it is not necessary to consider the shortest path problem of the ordinary road on the ground. 3. Midway track and route change Changing routes is something that happens during driving. This is due to the driver’s misunderstanding or misjudgment of the traffic instructions, and there may be deviations from the original route. At this point, the supervisory navigation system corrects the driver’s errors in time and tells him how to re select the route. As shown in Fig. 5.13, although the vehicle has left the expressway and entered the road, the navigation system will indicate on the display that it will re-enter the direction of the highway and its crossing. When the vehicle intelligent navigation is equipped with voice prompts at the same time, it will also give voice prompts to the driver. For example, before the driver showed the route shown in Fig. 5.12 (b) and strayed into the “Xiuzhou district” exit, the vehicle could only be diverted along the current ground from G320 to the “Su Jia” expressway.

192

5 Intelligent Vehicle Navigation and Traffic System

Fig. 5.12 Route displayed after clicking the destination

5.4 Vehicle Communication Technology in Intelligent Transportation

193

Fig. 5.13 Reselect path schematic after changing route

5.4 Vehicle Communication Technology in Intelligent Transportation As mentioned earlier, the intelligent transportation can not be separated from the wireless sensor network, and the vehicle unlimited sensing is based on the vehicle communication technology. In other words, ITS (Intelligent Transportation System) requires vehicles to be equipped with advanced communications equipment. Its purpose is to achieve communication between vehicles and traffic control centers, as well as communication between vehicles and vehicles, between vehicles and roadside ground service stations. It can be said that vehicle communication is an important technical to achieve intelligent transportation. From the current development trend, vehicle communication can be divided into wide area communication and short-range communication. Wide area communication is the main means to realize the communication between vehicle and control center. Wide area communication includes satellite communication, FM carrier communication, wireless digital public network and other means. Among them, FM carrier communication technology is relatively mature, can be used to send traffic information and other content. However, this communication mode has limited communication speed and is no longer suitable for the current digital communication. At present, the most popular mode of communication is the use of wireless digital public network as communication medium. For example, the GSM/GPRS/CDMA communication network, the digital two-way propagation rate can exceed 128 k. This communication technology has been quite mature and has a wide range of communication coverage. At the same time, the cost of communication is low, it has been widely used for anti-theft alarm, dispatching and navigation, and logistics

194

5 Intelligent Vehicle Navigation and Traffic System

transportation. Wireless digital networks have crossed 3G and 4G technologies, and will soon enter the new 5G technology field. In addition, satellite communication is a supplement to the two modes of communication mentioned above. In some areas where wireless digital public network and FM carrier communication cannot be covered, such as desert, oil field, plateau and other areas in Western China, satellite communication has become the preferred way of communication. The Beidou satellite communication system in China has already begun to provide special channels for vehicle communication. Many companies in China have established a special vehicle scheduling and navigation system based on the Beidou satellite communication. However, because of the high cost of this mode of communication, large-scale market promotion has been limited. With the establishment of low orbit geostationary small satellite network, the wireless communication technology of vehicle with the aid of satellite will become more popular. Compared with wide area communications, ITS’s dedicated short range communication is mainly used for communication between vehicles and vehicles, as well as between vehicle and road ground service stations. The communication protocol used in this environment is developed internationally, that is, the Dedicated Short Range Communication protocol (DSRC). DSRC is the basis of ITS, and it is a wireless communication system that connects vehicles and roads organically through two-way transmission of information. The system mainly consists of three parts: the vehicle unit OBU (On Board Unit), the roadside unit RSU (Road Side Unit) and the special short range communication protocol. The distance of short range communication is generally within 100 m, and the communication rate can reach 1 M. At present, it is widely used in the no parking electronic toll system (ETC).

5.4.1 The Combined Action Principle of GPRS and GPS In realizing automobile intelligent navigation, a large amount of traffic information is needed to interact in real time between the intelligent traffic and the vehicle navigation system. Therefore, it is necessary to give full play to the technical advantages of GPRS general wireless packet.

5.4.1.1

Introduction to GPRS Principle

1. GPRS main structure GPRS (General Packet Radio Service) is a wireless packet switching technology based on GSM system. It provides an end-to-end, wide area wireless IP connection. Generally speaking, GPRS is a high-speed data processing technology, the method is to use the form of “grouping” to transmit data to the user’s hand. Although GPRS is a transitional technology for the evolution of the existing GSM network to the third

5.4 Vehicle Communication Technology in Intelligent Transportation

195

generation mobile communication, it has a significant advantage in many aspects. The biggest advantage of GPRS is that its data transmission speed is far more than WAP. The transmission speed of the current GSM mobile communication network is 9.6 Kbps, and the GPRS receiver has reached the speed of 56 Kbps transmission. Moreover, the existing technology has been able to reach 115 Kbps (this speed is two times higher than that of the commonly used 56 K modem, or even higher). The main structure of GPRS communication is shown in Fig. 5.14. It includes: transmit/receive antenna, GPRS service support node, GPRS backbone network, gateway support node, the Internet and road network monitoring center server. On the basis of the original digital wireless communication network GSM, GPRS introduces two new network nodes: GPRS service support node and gateway support node. The GPRS service support node and the MCU are at the same level, and the memory unit of the single chip microcomputer is tracked to realize the security function and access control. It connects to the base station system through frame relay, gateway supports nodes to support interworking with external packet switching network, and supports connectivity through GPRS backbone network and GPRS service. The GPRS terminal obtains data from the customer system through the interface. The processed GPRS packet data is sent to the GSM base station. After the packet data is encapsulated by the GPRS service support node, the communication is carried out through the GPRS backbone and the gateway. The gateway supports the node to deal with the packet data and send it to the destination network, such as Internet or X.25 network. If the packet data is sent to another GPRS terminal, the data is sent to the GPRS service support node by the GPRS backbone network, and then sent to the GPRS terminal through the transmit/receive antenna. The GPRS communication body and each GPRS terminal constitute a complete GPRS communication system. 2. GPRS receiver features The GPRS receiver consists of single chip microcomputer, GPRS chip, SIM card, external interface and extended data memory. Among them, the single chip microcomputer supports the TCP/IP protocol, it is used to control the information receiving and sending of GPRS chip, and communicate with data through the standard RS232 serial port and external controller (such as the signal processor described in this book). At the same time, the software is used to realize the interruption, and then to complete the data forwarding. GPRS is a communication technology between the second generation and the third generation. The frequency band, frequency band width, burst structure, radio modulation standard, frequency modulation rule and TD-MA frame structure adopted by GPRS are the same as those of digital communication network GSM. Therefore, when building GPRS system on the basis of GSM system, most components of GSM system do not need to make hardware modification, only upgrading the corresponding software. With GPRS, the call setting time of the user is greatly shortened, and it can almost do “Forever online”. In addition, the cost of GPRS is based on the amount of data transmitted by the operators, rather than on the basis of the length of the connection time, so that the cost of service for each user is lower. The SIM card

196

5 Intelligent Vehicle Navigation and Traffic System Transmitting / receiving antenna

Internet

Service support node

Road network monitoring center server GPRS receiver

Backbone network

Gateway support node

Fig. 5.14 Schematic diagram of GPRS communication system

used by GPRS is called “user identification card”, it’s actually an intelligent card containing a large scale integrated circuit, it is used to register the important data and information of the user. Owning the SIM card means that it has the legal status of GSM communication, and the most important function of SIM card is authentication and encryption. When the user moves to the new area to dial or answer the phone, the switch must authenticate the user to determine whether it is a legitimate user. At the same time, the SIM card and the switch use authentication algorithm to calculate the authentication key and 8 bit random number at the same time. The SIM card is recognized when the result is the same, otherwise, the SIM card is rejected and the user can not call. SIM card can also use encryption algorithm, dialogue sound encryption, to prevent eavesdropping.

5.4.1.2

Structure and Principle of GPRS-GPS United System

1. GPRS-GPS joint system Architecture The GPRS-GPS joint system consists of four parts: GPRS-GPS unit, information acquisition unit, signal processing unit and ground control center, the structure of the system is shown in Fig. 5.15. The GPS module in the GPRS-GPS unit is used to receive the satellite positioning data, which can simultaneously display the position of the current vehicle in the electronic map at the same time on the display screen of the vehicle mounted LCD (Digital LCD) and the ground control center. The GPRS module in the GPRS-GPS unit is used to realize data interaction between the communication system on the vehicle and the Internet. These interactive information includes vehicle safety, vehicle failure, SMS and information, etc. 2. The working principle of GPRS-GPS united system First, the ground control center inputs the electronic map to the vehicle system through the internet.

5.4 Vehicle Communication Technology in Intelligent Transportation

197

Ground control center

Signal processing unit Control output Automobile working condition

Information acquisition unit

Fig. 5.15 Principle block diagram of GPRS-GPS combined system

In the navigation process, the on-board GPS receives the satellite positioning information in real time and transfers it to the signal processing unit through the GPRS. At the same time, GPRS can also access the traffic flow and road construction information through the internet. According to the current geographic position, vehicle driving condition and traffic information, the signal processing unit can determine the driving route, speed and other technical parameters of the vehicle after the data processing, analysis and decision-making. All the information received by the vehicle can be prompted to the driver in real time through the digital display (LCD), and the best driving control can be carried out through the servo structure of the vehicle and the traffic information. At the same time, the vehicle running state or fault condition can be sent to the ground control center through GPRS, so that the ground control center can real-time grasp the driving conditions of all the online vehicles and the traffic flow distribution of the whole road system.

5.4.2 The Role of 3G Technology in Intelligent Navigation As far as the current technical situation is concerned, the traditional GPS positioning means work well for the outdoor environment. However, for the indoor environment or where the satellite signals can not be covered, above the location of the vehicle, if there are no more than 3 satellites, the system can not be located from the “cold start” state. If we use wireless communication networks alone, we need to rely on the forwarding of signals from mobile stations (MS) to multiple base stations (BS). In addition, the communication network is affected by multiple signals, signal

198

5 Intelligent Vehicle Navigation and Traffic System

diffraction, weak signal and the geometry of the base station, and there will be a phenomenon of lower communication accuracy, poor availability and disengagement from the communication system. The main findings are as follows: in rural or suburban areas, due to the low density of communication sites, the performance and service availability of network solutions are poor, which makes it possible to make use of wireless communication network technology alone. However, in the rural or outskirts of the same remote area, the receiving effect of the GPS receiver is usually very good, which can receive signals of 4 or more satellites. Conversely, in dense urban areas and buildings, GPS receivers often cannot receive enough satellite signals to calculate location, but they can receive signals from multiple cell sites. So the better solution is to combine the complementary advantages of CDMA network and GPS. In this way at the same time, the availability, sensitivity and accuracy of the localization can be improved directly by using the cellular wireless network information and satellite based GPS information.

5.4.2.1

The Composition of Vehicle Navigation and Wireless Interconnection System Based on 3G Technology

1. Brief introduction of 3G technology The 3G (3rd Generation), which refers to the third generation of digital communication. The main difference between the third generation and the previous generation is the improvement of the transmission rate of voice and image data. Moreover, it can support at least 2 Mb/s, 384 kb/s and 144 kb/s transmission rate in indoor, outdoor and driving environments. It can handle many kinds of media signals, such as image, music, video stream, and provide various information services including web browsing, teleconference, e-commerce and so on, which is very suitable for transmitting real-time traffic conditions detected by video detectors installed on roads. With the successful application of the third generation mobile communication technology in the world, the Intelligent Vehicle Location and Navigation System (IVLNS) has had a profound impact. At the same time, 3G also provides a new technical method for the wireless Internet and the outside Internet, which has brought more security to the safety and comfort of the car. It can be predicted that if mobile communication means to the development of 3.5G, 4G and 5G, we can use the same technology to achieve the function of vehicle navigation and wireless interconnection, and it is more efficient than 3G based technology. 2. Composition of vehicle positioning and navigation system based on 3G The vehicle positioning and navigation system based on 3G is mainly composed of navigation system GPS satellite, vehicle GPS receiver, communication base station, mobile positioning center, electronic map database and server, as shown in Fig. 5.16, the virtual interconnect in the graph represents the wireless interconnection channel.

5.4 Vehicle Communication Technology in Intelligent Transportation

199

GPS satellite Content provider CDMA 3G network

Electronic map database

Mobile Location Center

Vehicular GPS receiver Web server Location server

Base station GPS reference equipment

Location entity Base station database

Internet

Fig. 5.16 Schematic diagram of vehicle navigation and wireless Internet

Vehicle navigation function must be realized by electronic map, but the electronic map stored in vehicle equipment can not meet the requirements of navigation, 3G technology can make the vehicle computer realize fast wireless access to the Internet whenever and wherever possible, the administrative area of the vehicles in the process of detection, can immediately through the wireless internet access Internet, then connect to the electronic map database in the area to download corresponding electronic map corresponding to download. This can not only reduce the cost of on-board equipment, but also make positioning and navigation more real-time and accurate. 3. Wireless interconnection function The wireless interconnection system based on 3G is mainly composed of vehicle equipment, communication base station, network and gateway equipment (PDSN) and Web server. Among them, PDSN (Packet Data Serving Node) is a gateway between wireless access network (RAN) and Packet Switched Public Data Network (PSPDN), that is, mobile terminals access internet access devices in CDMA2000 networks, PDSN is a critical device in the 3G program, without it, the unique functions of mobile terminals can not be realized. In addition to the function of downloading the electronic map, the vehicle can also access the Internet, send and receive e-mail anytime, anywhere, and do office work and entertainment on the internet.

5.4.2.2

Characteristics of on-Board Equipment Based on 3G Navigation and Interconnection System

1. System structure of vehicle navigation and wireless interconnection The hardware architecture of the vehicle navigation and wireless interconnection system is shown in Fig. 5.17. Some peripheral equipment, such as audio, display, control, storage and other equipment, which can be used as a special part of the navigation system, but also with other car entertainment equipment or non navigation

200

5 Intelligent Vehicle Navigation and Traffic System

Hands-free phone

Control device Storage device

Wireless access device Display device

Navigation and wireless interconnection embedded system

Sound equipment

Location and communication module

Fig. 5.17 Hardware structure of vehicle navigation and wireless interconnection system

Location and communication module

Hands-free phone

Hands-free phone control

Human-computer interaction interface

Wireless internet access

Embedded system of vehicle navigation and wireless interconnection

Geographic Information System Engine

Path planning

Path guidance

Map matching

Electronic map database

Fig. 5.18 Embedded system architecture for vehicle navigation and wireless interconnection

system in common, such as the display device can also be used to display the electric meter system for automobile signal display. The audio equipment can be shared with the TV, DVD and other systems. 2. Embedded system structure and module function The architecture of vehicle navigation and wireless interconnection embedded system is shown in Fig. 5.18. (1) Electronic map database The database is an essential part of the modern vehicle navigation system. It contains digital navigation maps stored in a predetermined format, providing a variety of important information, such as geographical features, road location and coordinates, traffic rules, infrastructure and so on. 3G navigation can download and update the local electronic map database in real time.

5.4 Vehicle Communication Technology in Intelligent Transportation

201

(2) Geographic information engine This is the interface of operation and query of electronic map database. It provides the functions of display, browse, dynamic refresh, zoom, rotate, drag and drop, interface switching, and related information retrieval and query service. (3) Positioning and wireless communication module This is a device based on Qualcomm 3G 1X MSM5100 chip. In addition to the function of the 3G system, the device also includes a GpsOne core that is tightly integrated with the chip integrated DSP. In addition, the Qualcomm MSM architecture uses the SnapTrack algorithm which supports multi-channel anti-aging and signal enhancement, which significantly improves the speed and sensitivity of the GPS signal acquisition, so that the system can correctly identify the vehicle’s current road section or the approaching intersection. Positioning and wireless communication module can further enhance the functions of the vehicle navigation system, through the CDMA 3G wireless communication network. Users between the vehicle and traffic management system can exchange real-time traffic information, making vehicle system and road network more safe and effective. The GpsOne kernel chip is a technology developed by the QUALCOMM company in the United States. When the GpsOne kernel chip is applied to the intelligent transportation system, it can combine the GPS positioning with the wireless communication network technology well. So it can achieve fast and accurate positioning of mobile devices. (4) Map matching module The module is used for positioning, comparison of road location information and location estimation of wireless communication module and the output map database, and through appropriate pattern matching and the recognition process to determine the road vehicle current and accurate position in the road, and provide a reliable basis for the realization of route guidance. (5) Path planning It can provide the best route for the driver to travel before or during the journey according to the traffic network information in the electronic map. At the same time, the navigation function based on 3G technology can also be used to obtain real-time traffic information from wireless communication network, and timely response to changes in road traffic conditions. (6) Path guidance According to the road information in the map database and the location of the current vehicle provided by the map matching module from the GpsOne positioning information, the appropriate real-time driving instructions are generated, in order to help the driver to travel along the scheduled route so as to reach the destination smoothly. (7) Human-computer interaction interface The human-computer interface provides an interactive interface between the driver and the vehicle computer system. It can input the operation instructions such as map display, information query, path planning, Internet, entertainment and so on into the computer system. The computer system can also return the

202

5 Intelligent Vehicle Navigation and Traffic System

information needed by users, such as vehicle location, optimal path planning result, real-time driving guidance command and so on, by voice prompt and visual graph. It can also display road image information that is detected by the camera device and transmitted through the 3G network, it can also return the network information to the display on the Internet.

5.5 Real Time Increment Method of Navigation Map No matter the American GPS, or the Chinese BDNSS (Bei-Dou Navigation Satellite System), or the Russian GLONASS, and the European Union GALIEO, all the electronic maps can not be separated. Completeness, accuracy and timeliness of the electronic map directly determines the function of navigation equipment. Technologies to ensure that the navigation information can be rapid accurate, and can be fully updated, have become the key elements of the satellite navigation technology. Navigation electronic map information is updated by a variety of means of data collection, access to the latest navigation electronic map data, so as to improve the reality and accuracy of navigation equipment. Generally speaking, there are two new methods for updating navigation electronic map data, including version update and incremental update. For version update, although the operation is simple, but the amount of data is large, it is difficult to apply wireless remote transmission (such as GPRS) downloading. For incremental updating, we only detect the change parts of navigation electronic map, such as adding, modifying, and deleting elements, get the changing elements and related information, and then form incremental files according to the change information. Therefore, the user can make data fusion (incremental fusion) based on the received incremental file and the original navigation electronic map data, and then form a new navigation electronic map data. It is obvious that the incremental update is better than the version update method, which can greatly reduce the time and cost of the user to update the map data. Incremental update improve the efficiency of update is the mainstream method of updating the current and future navigation electronic map data. However, how to efficiently and quickly obtain the increment of the electronic map and its integration with the original database is the key issue of the research. As an example of vehicle navigation, a novel approach to map increment is provided.

5.5.1 Information Incremental Updating Method Compared with ordinary electronic maps, the navigation map not only has map data, but also has the function of planning path. Satellite navigation is characterized by the ability to quickly know the location of its own location, based on the received satellite signals. Once the destination is set, it can automatically combine the satellite

5.5 Real Time Increment Method of Navigation Map

203

Fig. 5.19 Vehicle configuration image sensor example

signal with the map data to provide one or more paths for the driver to choose. When the road planning and construction changes, if the navigator can replan a route to the original destination according to the current information and its location, the navigator has the intelligent technical function. This intelligent navigator has two key technologies: (1) How to collect and fuse information increments. (2) After the increment of the electronic map, how to realize the re optimization of the travel path.

5.5.1.1

Acquisition of Information Increment

Visual sensing technology can be used to add image sensors to traffic tools (see Fig. 5.19). In Fig. 5.19, the image sensor adopts three pinhole cameras, which are located at the middle position of the upper edge of the forward looking windscreen frame, the middle position of the front cover of the engine and the position of the right rear mirror cover near the right edge. The three pinhole cameras are connected to the signal processor through three image signal channels, respectively. The signal processor is responsible for the processing, recognition and digital instruction output of the image signal. The output interface of the signal processor is connected to the serial input port of the navigator. The application of the technical methods described in the second chapter of this book can be realized as follows: (1) The memory of the road image information and the memory of the network nodes in the change point of the route. (2) Road speed limit identification and network node memory. (3) The geographic coordinates are used as the frame head code for each frame, so it is easy to link and fuse information.

204

5.5.1.2

5 Intelligent Vehicle Navigation and Traffic System

Fusion of Information Increment

The “information increment” refers to the addition, modification and deletion of information. The expression of “information increment” is actually a dynamic maintenance process of the “navigation path” topology relationship. At this point, the “complete association matrix” of the “navigation path” topology can be expressed as

x1 x A 2 .. . xM

⎡

y1 y2 . . . y N

⎤ a11 a12 . . . a1N ⎢ a21 a22 . . . a2N ⎥ ⎢ ⎥ ⎢ ⎥ .. . . .. ⎥ ⎢ .. ⎣ . . . ⎦ . a M1 a M2 . . . a M N

(5.8)

Defining X  {xi }  {x1 , x2 , . . . , x M } as “navigation path” topology node set, Y  y j  {y1 , y2 , . . . , y N } is the “navigation path” topology edge set. The “complete association matrix” can also be expressed as   A  ai j M×N

(5.9)

Among them, ai j ∈ {−1, 0, +1}, +, and - symbols represent positive and negative two directions, respectively. When the edge of a topology is assigned to a distance (km) as a weight, the elements in the set of edges represent the distance of the actual road. At this point, the “complete association matrix” is mapped to ⎤ ⎡ y1 0 . . . 0 ⎥ ⎢ ⎢ 0 y2 . . . 0 ⎥ ⎥ ⎢ ˙  A⎢ (5.10) A ⎥ .. ⎥ ⎢ ⎣0 0 . 0 ⎦ 0 0 . . . yN N ×N

The incremental fusion process is: (1) The increase of road information When the “new road” appears, it involves the increase of the road topology node and (or) edge,   association matrix”   is extended. At   that is, the “complete this time, A  ai j (M+1)×N or A  ai j M×(N +1) , or A  ai j (M+1)×(N +1) (2) Modification of road information Once the “road change” appears, it involves the modification of road information. That is, the changes in the specific values of the element ai j in the A, and the change of the specific element yk in the topology edge set Y, k ⊆ j.

5.5 Real Time Increment Method of Navigation Map

205

(3) Deletion of road information This means that a road is closed or “lost”, that  is, the “complete   association matrix” is compressed. At this time, A  ai j (M−1)×N or A  ai j M×(N −1) ,   or A  ai j (M−1)×(N −1) 5.5.1.3

Optimization of Information Fusion

In general, path optimization can be divided into two types: the shortest distance path and the most proficient time path. (1) Shortest distance path Order, Bk(l) is a set of associated elements of a point k to point l connection path

Bk(l)  a˙ kp , . . . , a˙ lq

(5.11)

In the form, k, l ⊆ i, p, q ⊆ j, a˙ kp , . . . , a˙ lq is the element in the “fully associated ˙ that is, matrix mapping”A,

a˙ kp , . . . , a˙ lq ⊆ ai j

(5.12)

Also, Ck(l) is the total set of the total length of the connection point k to point l path made up of associated elements set. Ck(l)  {r }

(5.13)

In the form, r  1, 2, . . . Then, the shortest path Cmin is the minimum set of Ck(l) , that is Cmin  minCk(l)  min{r }

(5.14)

When the new increment of the map information appears, Cmin can be obtained ˙ by the fusion of “complete association matrix mapping”A. (2) The shortest path of time The expression of the edge diagonal matrix element in formula (5.10) is converted from “distance” to “time”. That is ⎡ ⎢ ⎢ ⎢ ˙  A⎢ A ⎢ ⎢ ⎣

 y1 v1 0

...

0

y2 v2 . . .

0

0 

0

0

0

0

..

.

0  . . . yN vN

⎤ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎦

(5.15)

N ×N

206

5 Intelligent Vehicle Navigation and Traffic System Driving coordinate tracking

No

Yes

Collecting images and storing

Change of road condition?

No Memory and identification of geographical coordinates

New road?

No

The road is modified?

Yes Draw new road section

Yes

Yes

The road is closed? No

Delete the associated edge

Point and edge correction

Memory and identification of new nodes

Re optimizing the path

Fig. 5.20 Road information memory and dynamic identification control

In the formula, vi is the corresponding edge (Road) rated speed (i.e. maximum speed), from the original map data (if any) and road speed limiting random information acquisition and recognition (no road speed limit sign case, take the road speed limit is 30 km/h), i  1, 2, . . . , N . Then, the operation of the repeated formula (5.11)–(5.14) is repeated.

5.5.1.4

Dynamic Memory and Identification of Road Conditions

Once the vehicle is configured with the image sensor and its signal processing system, when it is running along a predetermined path, it can change his route according to the road conditions, collecting the real-time traffic scene, and automatically to realize memory and identifies those new added network nodes and edges (or modify, or delete), and update the local network topology relation (see Fig. 5.20).

5.5.2 Application Example Taking a driving route in Shanghai as an example, the following steps are as follows: (1) Setting up the destination (see Fig. 5.21): starting from a place in Shanghai to drive to the Minhang campus of Shanghai Jiao Tong University.

5.5 Real Time Increment Method of Navigation Map

207

Fig. 5.21 Example from start to end point navigation

Fig. 5.22 Record speed limit signs and geographical coordinates

(2) Along the way, the navigation system will record the speed limit mark in real time and identify it (see Fig. 5.22). (3) On the way to “Jin Du Xi Road”, when the new road “Huaxi road” was opened, the vehicles were scarce and decided to change the road route (see Fig. 5.23). (4) Because the original map does not have this road record, “navigation” temporarily appears “malfunction” phenomenon, then, the navigation device automatically supplements the information (see Fig. 5.24).

208

5 Intelligent Vehicle Navigation and Traffic System

Fig. 5.23 Actively changing driving route examples

Fig. 5.24 Information is automatically added

Fig. 5.25 Passively changing lanes

(5) Traveling to Guanghua Road, the road ahead is closed. (see Fig. 5.25), left turn into Guanghua Road and drive eastward, record the new node and re optimize the path. (6) A few days later, when moving through the road again, the navigation system will automatically select the latest optimization route for navigation (see Fig. 5.26). The proposed incremental real-time update algorithm has the following advantages compared with the same technology.

5.5 Real Time Increment Method of Navigation Map

209

Fig. 5.26 Example of the re optimization of the path

(1) With the incremental real-time updating algorithm, the navigation device can automatically capture and recognize the external scene (including: road conditions and speed limit signs, etc.) through the external image sensor and its signal processor. At the same time, it can take the geographic coordinate data as the image header code, so it is convenient for information linking and fusion. (2) The increment of road information (including adding, modifying and deleting) of this technology realizes real-time updating and saving. Because of its relatively small update data and small memory footprint, the read and write rate is high. (3) The technology also has the function of automatically remembering, marking, updating and optimizing the running route.

Chapter 6

Vehicle Auxiliary Intelligent Technology

Automobile auxiliary intelligent technology mainly includes: car light intelligent control, intelligent door switch, vehicle wiper intelligent control and other technologies.

6.1 Intelligent Control Technology of Car Light The vehicle visual sense system can be used directly in the intelligent control of the vehicle lamp. This technique has been realized by using an environment image collected by any of the pinhole camera 1–5 and after the intelligent operation.

6.1.1 System and Circuit Structure 6.1.1.1

System Overview

The whole system consists of three parts: signal processor, controller and power electronic switch circuit (referred to as a switch circuit). The schematic diagram of the system has been shown in Fig. 6.1. In Fig. 6.1, the signal processor includes: image input interface, analog to digital conversion module, digital image processing module and output interface. The digital image processing module is the core module of the signal processor, and it undertakes the processing, analysis and decision of the image information. The controller is responsible for the interpretation of the output instruction of the signal processor and the expression of the control strategy, so the optimal control of the headlights, taillights and anti-fog lamps (The lamps that dispels the fog) can be achieved. The optimal control system here refers to the maximum power saving and energy saving while ensuring safe driving. © Shanghai Jiao Tong University Press, Shanghai and Springer Nature Singapore Pte Ltd. 2019 X. Zhang and M. M. Khan, Principles of Intelligent Automobiles, https://doi.org/10.1007/978-981-13-2484-0_6

211

212

6 Vehicle Auxiliary Intelligent Technology Signal processor

Controller

Headlamp

Switch circuit 1

Pinhole camera Taillight

Switch circuit 2

Fog lamp

Switch circuit 3

Fig. 6.1 Hardware structure block diagram of automobile light intelligent control system DC-DC switching and isolation DC input

High frequency rectifier filter

Base driver

PWM controller

Controlled DC output

Error amplifier

Vout

Reference voltage

Fig. 6.2 Topological structure of vehicular switch circuit

The switch circuit 1 is composed of IGBT power electronic switch device. Under the premise of ensuring zero voltage switching and zero current turn off, it can achieve the optimal duty cycle output by adjusting the pulse width PWM (Pulse Width Modulation) control strategy. Therefore, the highest energy efficiency of the illuminance of the headlamp is achieved. The switch circuit 2 and 3 only need to open and disconnect the tail lamp and the fog lamp respectively.

6.1.1.2

Switch Circuit Structure

1. Topological structure In general, the topology of the switch circuit configured on the vehicle is shown in Fig. 6.2. In Fig. 6.2, the DC-DC transform uses pulse width modulation (PWM) technology to adjust the output voltage by adjusting the pulse width, and the pulse width can also be adjusted by adjusting the input voltage, so the voltage regulator range is wide. The so-called pulse width modulation (PWM), that is, by adjusting the pulse width to adjust the output voltage. Pulse width modulation has obvious advantages and occupies the leading position in the DC-DC transform.

6.1 Intelligent Control Technology of Car Light Fig. 6.3 Buck principle circuit

213

L

Q D

Automobile headlamp

C

2. Basic circuit types of DC-DC The basic circuit of DC-DC transform includes: Buck, Boost, Buck-Boost and Cuk and so on. (1) Buck circuit is also called series switching voltage stabilizing circuit or buck chopper circuit. The schematic diagram of the Buck converter is shown in Fig. 6.3. The Buck circuit has two basic operating modes, namely, the continuous inductor current mode and the discontinuous inductor current mode. The continuous inductor current means that the output filter inductor current is always greater than zero, and the inductor current interruption means that the inductor current is zero during the period when the switch is switched off. There is a critical state between the two states, that is, the inductor current is just zero at the end of the switching off of the switch. When the inductor current is continuous, there are two kinds of switching states in the Buck converter. When the inductor current is discontinuous, there are three kinds of switching states in the Buck converter. (2) The Boost circuit is a boost chopper circuit. Like the Buck converter, the Boost converter also has two operating modes: continuous inductor current and discontinuous current. When the inductor current is continuous, there are two kinds of switching states. When the inductor current is interrupted, there are three kinds of switching states. (3) The Buck-Boost circuit can work in the Buck as well as Boost mode. The polarity of the input voltage is opposite to the polarity of the output voltage. In the Buck and Boost converters, the input is positive, and output side is negative, there is an energy flow from the power supply to the load process, in the BuckBoost converter, the energy is stored in the inductor first, and then the energy is released from the inductor to the load. (4) The Cuk circuit is a kind of improved circuit which aims at the shortcoming that the inductance L of Buck-Boost converter has big fluctuation to the input and output current. The converter uses two inductors, one at the input and one at the output, thus reducing the current ripple. Usually, the Buck circuit is used for the switch control circuit of the vehicle lighting. The DC-DC converter circuit with Buck inductor current discontinuous operation is more commonly used. When the square wave control signal is added to the control pole of the power electronic switch device Q, the power electronic switch device is switched periodically under the excitation of the control signal. As shown in

214

6 Vehicle Auxiliary Intelligent Technology

Fig. 6.4 Characteristics of Buck circuit with inductor current interruption

Fig. 6.4, the VG E is the square wave control signal output voltage, the T is the control signal cycle, and the ton is the turn-on time of the power electronic switch device, to f f is the turn-off time of power electronic switching devices, i L and i C are inductance and capacitance currents, respectively. Q is the charge accumulation on the capacitor C during the turn-on time of the power electronic switching device. Whether the current i L in the inductor is continuous depends on the switching frequency, the filter inductance and the capacitance value. Figure 6.4 shows the current and voltage conversion process, that is, the three-working state: (1) When Q conduction, the inductance current i L increases from zero to the maximum value i L max , at this time the inductance voltage u L is uL  L

di L dt

(6.1)

Its linear approximate expression is Vd − Vo  L

i L t

(6.2)

Obviously, t  ton , therefore, the inductance current increment i L+ is i L+  i L |opened 

Vd − Vo ton L

(6.3)

6.1 Intelligent Control Technology of Car Light

215

Type medium, Vd is the voltage output after the time t, and Vo is the mean of the output voltage of the converter circuit. (2) Q turn off, diode D freewheeling, i L down from i L max to zero, at this point, the inductance current is reduced to i L− i L−  i L |closed 

 Vo  ton + to f f L

(6.4)

(3) Q and D were cut-off, and i L remained zero during this period, the load current is supplied by the output filter capacitor. According to the principle of conservation of energy, the relationship is as follows i L+  i L−

(6.5)

Therefore, it can be deduced Vo 

ton to f f

2 tton + 1

Vd 

δ Vd 2δ + 1

(6.6)

of f

Type medium, δ  tton , named as the duty ratio of the inverter chopper. of f From the formula (6.6), it can be seen that the average voltage Vo output of the DC-DC converter is related to the duty cycle δ and the terminal voltage Vd which is reflected by the load current. 3. PWM control principle The duty ratio control of an inverter chopper realized by PWM. PWM can be divided into two control modes: voltage type and current type. (1) Voltage type PWM The voltage mode pulse width modulator is a voltage pulse converter. The principle of the pulse width modulator using sawtooth wave as the modulation signal is shown in Fig. 6.5. Among them, Fig. 6.5a is the working principle block diagram of PWM, Fig. 6.5b is the schematic diagram of PWM generation. (Note: for more detailed explanations, readers can refer to the power electronics professional literature). It can be seen from Fig. 6.5b, after the voltage control Vcontr ol is modulated by the sawtooth modulation signal V , the output PWM switch signal is the same frequency as the sawtooth wave signal, but its pulse width is proportional to the size of the Vcontr ol pulse width modulation signal, that is, PWM output. (2) Current mode PWM The current mode PWM controller has many advantages compared with the traditional voltage type PWM controller with only output voltage feedback. From the

216

6 Vehicle Auxiliary Intelligent Technology

(a)

Vset

Reference voltage (Desired output voltage) Actual output voltage (Feedback voltage)

Vo

Control voltage Error amplifier

Comparator

Vcontrol

PWM output

Sawtooth wave generator

(b)

Fig. 6.5 Principle diagram of pulse width modulator

circuit structure point of view, an inductor current feedback is added. And the current feedback is used as the PWM’s oblique wave function, it no longer needs the sawtooth wave (or triangular wave) generator, more importantly, the inductor current feedback can make that the performance of the system has more obvious superiority. As shown in Fig. 6.6, it is a constant current PWM controller. The control circuit is double loop control, which has voltage outer loop and current inner loop, and the peak current is in the inner loop. The feedback current in the current inner loop is the inductor current or the switching current. Rs is the sampling resistor of feedback current. For the illumination control of the automobile headlamp, any type of voltage or current type PWM can be selected. Reference voltage (Desired output voltage)

Vset

Actual output voltage (Feedback voltage)

Vo

Error amplifier

Ve

Adder

Comparator

Rs

PWM output

iL

Oblique wave compensating signal

Fig. 6.6 Schematic diagram of current mode PWM controller with constant frequency

6.1 Intelligent Control Technology of Car Light

217

When the control voltage signal VG E of the IGBT control stage is a pulse voltage signal which appears according to the PWM regulation law, the DC voltage can be chopped by the pulse voltage. While, the IGBT is conducted, and the chopper duty cycle is proportional to the control voltage Vcontr ol . Finally, the output voltage of the DC-DC converter made up of IGBT is an equivalent DC voltage that follows the change of the control voltage Vcontr ol until the voltage at the two ends of the headlamp is equal to the reference voltage. In other words, the PWM control signal output by the PWM controller will change the voltage output of the supplied headlight with the “intent” of the reference voltage. Therefore, headlights can provide the driver with the clearest and brightest lighting in front of the vehicle, and it can also achieve the best energy-saving effect.

6.1.2 The Generation Principle of Controlling the Reference Voltage of Headlamp The reference voltage is the basis for the illumination control of the automobile headlamp. This basis comes from the intelligent detection results of road environment illumination by vehicle pinhole camera.

6.1.2.1

Detection of Environmental Illuminance

The environmental illuminance is characterized by the mean value of the image gray, that is, the mean of the environmental gray level is one-to-one correspondence with the ambient illuminance. Assume, the ambient illumination is L 1 , and the corresponding gray value of the environmental image is , and the additional illuminance of the lamp is L 2 . Usually, there is a nonlinear relationship between the environmental illuminance and the mean value of the image gray, can be expressed as  L 1  f 1 () (6.7) L 2  f 2 () The geometric meaning is shown in Fig. 6.7. It is assumed that the image with W × H is collected in real time, W and H are the column values and row values of image pixels, and the average gray value of environment image is  

W  H  i1 j1

f (i, j)

(6.8)

218

6 Vehicle Auxiliary Intelligent Technology

In the formula, f (·) is the gray value of the pixel on the corresponding coordinate point. The actual effect of illuminance in Fig. 6.7 represents the actual effect of the illumination of the environment after the illumination of the lamp is added. “Actual effect illuminance” depends on two factors: (1) The additional illuminance of the headlight should consider the illuminance of the environment, but the illuminance of the two cannot be simply added. (2) Different environments should follow the illumination science and standard.

6.1.2.2

Reference Voltage Generation Process

The concrete steps are as follows: 1. Obtain the gray mean  of the environment image by collecting the current environment image. 2. According to the gray mean  of the environmental image, the value of the current environmental illuminance L 1 and the supplemental illuminance L 2 of the headlamp are obtained using establishing the look-up table by means of the illuminance and the mean value curve of the image (see Fig. 6.7). 3. Identification of environmental types, such as highways, tunnels, or urban roads, by vehicle visual sense system, and determine the standard that environmental illuminance should be mastered by the lighting reference table (see Table 6.1). 4. Determine the reference voltage Vset

Actual equivalent illuminance

L

Ambient illuminance

L1

Supplemental illuminance

L2

Γ Fig. 6.7 Relationship between gray level and illuminance of environmental image Table 6.1 Reference table of road illuminance Road type Expressway, tunnel In tunnel entrance and exit Illumination reference/LUX

L a ≥ 150

L t ≈ 70

Downtown area Residential quarters L c < 45

L r < 15

6.1 Intelligent Control Technology of Car Light

219

(1) When the car is driving on the highway or at the entrance of the tunnel, if L 1 + L 2 ≤ L a , then Vset  K



La − L1

(6.9)

In the formula, K is the coefficient of proportionality, which is determined by experiment. For example, take L a  150 (LUX), there is  Vset  K 150 − L 1

(6.10)

If L 1 + L 2 > L a , then Vset  K



L1

(6.11)

(2) When the car is driving in the tunnel, if L 1 + L 2 ≤ L t , then Vset  K



Lt − L1

(6.12)

For example, take L t  70 (LUX), there is  Vset  K 70 − L 1

(6.13)

If L 1 + L 2 > L t , there are two kinds of cases: (a) when L 1 ≥ L t , then Vset  0. (b) When L 1 < L t , the operation is still performed according to formula (6.12). (3) When the car is driving in the busy urban area, if L 1 + L 2 ≤ L c , then Vset  K



Lc − L1

(6.14)

 Vset  K 20 − L 1

(6.15)

For example, take L c  20, there is

If L 1 + L 2 > L c , there are two kinds of cases: (a) When L 1 ≥ L c , then Vset  0. (b) When L 1 < L c , the operation is still performed according to formula (6.14). (4) When the car is driving in the residential area, if L 1 + L 2 ≤ L r , then Vset  K



Lr − L1

(6.16)

 Vset  K 5 − L 1

(6.17)

For example, take L c  5, there is

220

6 Vehicle Auxiliary Intelligent Technology

If L 1 + L 2 > L r , there are two kinds of cases: (a) when L 1 ≥ L r , then Vset  0. (b) When L 1 < L r , the operation is still performed according to formula (6.16). It can be seen that the vehicular visual sensing system can determine whether the headlights need to be opened according to the natural illuminance intelligence of the environment and transmit them to the PWM controller through a given reference voltage. By adjusting the duty cycle of DC voltage, the PWM controller can accurately control the voltage at both ends of the automobile headlamp electrode, so that the best light intensity (power) can be emitted from the car headlamp. Ultimately achieve: not only to ensure safe driving, but also the pursuit of the energy-saving effect of the car.

6.1.3 Intelligent Control of Anti Fog Lamp and Taillight 6.1.3.1

Intelligent Control of Anti Fog Lamp

The image of the vehicle in front of the car is captured by a pinhole camera on the vehicle visual sense system, by analyzing the image of the scene to determine whether to open the anti-fog lamp. 1. Determine the decision threshold S0 S0 is the basic point to judge the environment under the conditions of rain and fog, the specific methods are as follows: (1) Collect different degrees of foggy image sets { f k (i, j)}, k  1, 2, . . . (2) After intensifying the fog  image,two-valued processing is carried out to obtain two-valued image sets fˆk (i, j) , k  1, 2, . . . (3) Find the boundary points in the two-valued image fˆk (i, j), and do the boundary point statistics, and use Nk to represent the number of fˆk (i, j) boundary points corresponding to the k two-valued image. (4) Determine the value of S0 , that is S0  max{Nk }

(6.18)

2. Anti-fog lamp opening and closing policy (1) After the scene image is enhanced, the two-value processing of the image is processed. (2) Find the boundary points of the two-valued map of the site and carry out the statistics of the boundary points to obtain the boundary point value N ∗ . (3) Control decision: when N ∗ ≤ S0 , turn on fog lamp; otherwise, turn off fog lamp.

6.1 Intelligent Control Technology of Car Light

221

Judge foggy weather by image, there are many ways to do it. For example, you can also calculate the gray histogram of the image to judge. If the grayscale histogram, the gray number ˆ of the most pixels is very close to the gray mean , at the same time, the pixel number M close to the gray mean  is close to the total pixel number W × H , that is ⎧ ˆ ⎨  −  ≤ ε1 (6.19) ⎩W × H − M ≤ ε 2

However, it can be concluded that the current driving environment has been shrouded in fog. The ε1 and ε2 in formula (6.18) are the threshold values of pixel gray and pixel number respectively. Once the fog is judged, the signal processor outputs the control command to the fog lamp switch circuit through the output interface, after the A/D conversion and voltage amplification, trigger the control pole of the electronic switch to switch on the electronic switch, anti-fog lamp, and the tail light. The tail lamp and the anti fog-lamp are in the state of movement, when the anti-fog lamp is opened, the tail lamp must be turned on.

6.1.3.2

Intelligent Control of Taillight

Taillight control is relatively simple. In addition to the anti-fog lamp and the headlamp follow the relationship, it is mainly based on the environmental illuminance to determine whether to open or close. (1) According to the decision threshold 0 of the environmental illuminance, the gray mean value  is obtained by the formula (6.8). (2) When  ≤ 0 , it is determined that the environment is not enough illuminance, the taillight should be opened. The typical taillight control principle is shown in Fig. 6.8. In the diagram, the controller only needs to convert the output instruction of the signal processor into digital to analog converter, and then directly drive the control pole of the switch circuit by voltage amplification. The switch circuit can directly adopt bidirectional thyristor to realize the opening and closing of the taillight.

6.2 Intelligent Switch Technology of Car Door To prevent the theft of vehicles, the reliable switch of the door becomes an important technology that the vehicle manufacturers cannot ignore. Regarding the current technical situation, although there are many kinds of door switches, most of them use mechanical or electronic cryptography technology. It has been reported that because

222

6 Vehicle Auxiliary Intelligent Technology

Pinhole camera Image processing

Γ

Comparator

Control instructions for fog lamps Headlamp control command

Γ0 Signal processor

+

Controller

Switch circuit

Taillight

Fig. 6.8 Principle diagram of taillight intelligent control

Pinhole camera

Image processing Fingerprint processing

Semiconductor fingerprint sensor

Signal processor

Decision operation

Controller

Lock

Fig. 6.9 Basic structure of intelligent car door lock hardware

the password of the door lock is cracked, the stolen articles or stolen vehicles are common. Therefore, owning a set of intelligent switch doors is increasingly becoming a reality that people want. There are many kinds of intelligent switch technology, which is difficult to explain in detail. This paper mainly describes a kind of intelligent lock technology with higher intelligence level, that is face recognition and fingerprint recognition lock.

6.2.1 System Constitution The key hardware of the system consists of the image sensor (pinhole camera), semiconductor fingerprint sensor and signal processor, etc. 6.2.1.1

System Composition and Sensor Settings

The basic composition of the face and fingerprint recognition lock hardware is shown in Fig. 6.9. The signal processor in the graph can be a single chip (such as DSP), can also be composed of multiple CPU, such as DSP chip to assume two functions of image processing and decision calculation, and the fingerprint processing is undertaken by ARM chip alone. The pinhole camera and the semiconductor fingerprint sensor are installed on the left side of the car body. The former can be installed on the left side of the door frame of the cab, and the latter is directly installed on the handle of the left door of the cab, as shown in Fig. 6.10.

6.2 Intelligent Switch Technology of Car Door

Semiconductor fingerprint sensor

223

Pinhole camera

Fig. 6.10 Schematic diagram of sensor setting mode

By the way, fingerprint sensors are mainly divided into two categories: optical fingerprint sensor and semiconductor fingerprint sensor. (1) Optical fingerprint sensor mainly uses the principle of light folding and reflection, light shoots from the bottom to the three prisms, the light beams through the prism, the light is on the uneven fingerprint, refraction and reflection of the brightness of the light will be different. CMOS or CCD image sensor is used to collect the picture information of different light and shade levels, and then the fingerprint collection is completed. The fingerprint sensor is not suitable for intelligent car door lock. (2) Semiconductor fingerprint sensor, includes either capacitive or inductive, the principle is similar, namely large-scale semiconductor unit integrated into a “flat” contact sensor chip, when the finger is attached to the form with the capacitor (inductor) on the other side, because the finger plane irregularity and actual distance the convex and concave point contact plate is not the same, capacitance or inductance numerical form is not the same, the sensor will according to different values of capacitance or inductance parameters of this principle will be the value of the series, so the data collection on fingerprint. Semiconductor fingerprint sensor is most suitable for intelligent door lock sensor, such as: AT77C104A FingerChip, which is a sliding fingerprint sensor (sweep fingerprint sensor). Its advantages are: small size, only 1.5 × 15 mm, strong robustness, the adjacent fingerprint frame has no rotation deformation, low power consumption, the image acquisition current is 4.5 mA, transmission current is 1.5 mA, the sleep mode current (standby state) is less than 10 µA.

6.2.1.2

Structure Features of Fingerprint Processing Subsystem

In general, fingerprint processing subsystem consists of operation processing module and fingerprint sensing module. A fingerprint processing subsystem consisting of ARM9 chip AT91RM9200 and sliding fingerprint sensor AT77104A FingerChip is taken as an example, its structure features are: the operation speed is fast (when the

224

6 Vehicle Auxiliary Intelligent Technology

AT77104A FingerChip

CDRAM×2 Fingerprint processing subsystem

AT91RM9200

DATAFLASH

DSP

Controller

Lock

Fig. 6.11 Structure principle of fingerprint processing subsystem

operating frequency is 180 MHz, the operation speed is 200 MIPS), low power consumption, it can provide on-chip or off-chip memory as well as a series of peripheral control, communication and data storage can be flexibly configured. The frequency of work and the robustness of the working environment have major advantages. 1. Characteristics of processing subsystem Using 32-bit processor AT91RM9200 to build an embedded fingerprint processing subsystem. The system uses two pieces of 16-bit SDRAM to configure into 32-bit width high-performance memory, read data with four bytes as a unit, thus speeding up the data reading speed. At the same time, 8 M of DataFlash is extended to store Uboot, Linux file systems and applications. The communication process of the system is: (1) The communication between the host and the ARM board includes: firstly, the PC host sends the file RomBoot.bin to the AT91RM9200 built-in ROM by using the Xmodem protocol in the super terminal, and automatically runs after the download. Secondly, the RomBoot.bin and U-Boot.bin programs are downloaded and stored to DataFlash respectively, and the U- Boot is automatically started after reset. Finally, the Linux mirror file and the application program are downloaded to DataFlash through the Ethernet port. Once reset, the development board enters the Linux system. (2) SPI interface completes the communication between AT77C104A and control chip. The control chipsets the working mode of AT77C104A by writing registers. AT77C104A transfers the collected data to SDRAM. (3) In the embedded system, the fingerprint frame sequence is spliced, and the spliced fingerprint image is exported through the USB interface. The basic structure of the fingerprint processing subsystem is illustrated in Fig. 6.11.

6.2 Intelligent Switch Technology of Car Door

225

Fig. 6.12 Sensor signal processing connection diagram

2. The communication process between AT91RM9200 and AT77C104B FingerChip The fingerprint sensor chip AT77C104A through FingerChip temperature sensor array over the fingerprint ridges and valleys to obtain fingerprint data. The chip provides the SPI interface, and has two communication buses: (1) SLOW bus: corresponding to the SLOW mode, control, control and read/write internal registers. (2) FAST bus: corresponding to the FAST mode, used to obtain pixels, so that the host gets all the fingerprint pixels. In the fingerprint processing subsystem, the SSC interface of AT91RM9200 relates to AT77C104B FingerChip. SSC contains separate receivers, transmitters, and a clock divider. Each transmitter and receiver have three interfaces: the TD/RD signal for data, the TK/RK signal for clock and the TF/RF signal for frame synchronization. When the AT91RM9200 communicates with the AT77C104B FingerChip, the former is in the host mode, the latter in the slave mode, and the connection mode is shown in Fig. 6.12. In the communication process, the clock RK of the SSC receiver is driven by TK, and the receiving signal is synchronized with the sending end signal, so the TF is connected to the RF. AT91RM9200 through the I/O port (PIO_PA5) provides a chip select signal, select the working mode of the fingerprint sensor. SSC programmable high level and two 32-bit dedicated PDC channels can be used for continuous highspeed data transmission without processor interference, which is suitable for fast acquisition of fingerprint data. AT77C104A FingerChip has 13 registers inside. AT91RM9200 writes internal mode registers through AT77C104A FingerChip, set FingerChip to get the pattern of pixels. At this point, AT91RM9200 sets the FSS (Fast, SPI, Slave, Select, low level) signal of FingerChip to low level by PIO_PA5. After setting, AT91RM9200 is the host, and FingerChip is slave. The MISO signal of FingerChip inputs the data collected to the RD port corresponding to the SSC port of AT91RM9200 and stores it into the SDRAM.

226

6 Vehicle Auxiliary Intelligent Technology Standby state Fingerprint identification Face recognition Yes Car owners? Yes Car owners? No

Door unlocks

No Keep the door closed

Fig. 6.13 Flowchart of master control program for intelligent car door lock

Each pixel obtained by the fingerprint sensor is represented by a 16 Decimal System number, corresponding to 4 clock cycles. When the sensor is transmitted through the SPI port to obtain a frame of data, a frame synchronization signal F0F00200 is transmitted, and then the fingerprint data with 232 × 8 pixels is transmitted. Therefore, it takes n  (232 × 8 + 8) × 4  7496 clock cycles to transmit one frame of data each time. When FingerChip works with 6 Mbps, 804 frames of fingerprint data can be obtained per second. The obtained fingerprint data are stored in SDRAM, and the fingerprint sequence is spliced into complete fingerprint images by fingerprint stitching program, and then transmitted to the DSP main processor through the communication interface. It must be pointed out that the structure of fingerprint identification subsystem and the communication principle between chips will develop with the development of chip technology. When readers get and read this book, the chip technology must have a new technology. But the basic principles still haven’t lost its guiding significance.

6.2.1.3

System Working Principle

To enhance the reliability and confidence redundancy of the car door lock, the system adopts the double recognition algorithm of the car owner’s face image and fingerprint. After the system confirms that the person who tries to open the door is actually the owner, the door can be opened. When the owner (or others) approach the door, the system begins to collect, process and analyze the current human face. When the owner (or others) hands hold the handle of the door, the thumb must be pressed on the induction surface of the fingerprint sensor. Only when the information of the two types of sensing (face image sensing and fingerprint sensing) is authenticated by the system, the door can be automatically opened (or easily opened). Otherwise, as long as there is a kind of sensor information does not meet the certification standards of the system, the door cannot be opened. The main program flow of the system is shown in Fig. 6.13.

6.2 Intelligent Switch Technology of Car Door

227

6.2.2 Kernel Algorithm of Intelligent Door Switch The key algorithms of intelligent door switch mainly include: face recognition algorithm and fingerprint recognition algorithm.

6.2.2.1

Face Recognition Algorithm

Face recognition using principal component analysis (PCA) method. The specific process includes: importing the training sample set of the face, calculating the training sample eigenvalue and eigenvector, importing the face test sample set, calculating the image feature vector and classification recognition. Since, the face recognition is a matching problem of 2D projection images of 3D plastic objects. Its difficulty lies: (1) Facial plastic deformation (such as facial expressions, etc.) has uncertainty. (2) Face patterns are diverse (such as beard, hairstyle, glasses, make-up, etc.). (3) There are also uncertainties in the process of image acquisition (such as intensity of illumination, the direction of light source, etc.). Face recognition is mainly based on the characteristics of the face, according to the large differences among different individuals, and for the same person, there are relatively stable features of the metric. The recognition process can be divided into the following steps: (1) Face image preprocessing Before face feature extraction and classification, it is necessary to do geometric normalization and grayscale normalization. Geometric normalization means that the human face is transformed to the same location and the same size according to the location results of the face. Gray normalization refers to the compensation of the image illumination, and the illumination compensation can overcome the influence of illumination change to a certain extent and improve the recognition rate. (2) Import human face training samples The face training samples are imported into each two-dimensional face image data and transformed into a one-dimensional vector. For different facial expression images, a certain number of images are selected to form training sets. Assuming that the size of the image is u × v (u and v are row and column prime numbers), the number of face samples used for training is n, let m  u × v, then the training set is a matrix with m × n. Import the image of the owner (one or more people together), take Fig. 6.14 as an example, if this is the owner of a car, the owner’s face category can be divided into two categories: Car owners are the first category, training samples. Non-car owners are the second category, test samples. A total of 9 images were imported from the driver’s face training sample, each original image has 256 gray levels.

228

6 Vehicle Auxiliary Intelligent Technology

Fig. 6.14 Legends of a car owner’s face database

In the graph, the ith image of first kinds of human faces can be represented as one-dimension vector xi(1) T T

(1) (1) (1) xi(1)  xi1  xi(1) xi2 . . . xim j

(6.20)

In the form, i  1, 2, . . . , n face samples, for the legend in Fig. 6.14, each type of face training samples takes 9 photos, namely i  1, 2, . . . , 9. n is the first type of face sample number. j  1, 2, . . . , m is the number of pixels taken from each sample image. The so-called face sample, that is, a specific class (person) each image can contain changes in expression, gesture, and scale, and the attitude of micro change, scale changes within 20% of the attributes of the image. When each image is u  112 and v  92, then m  10304 represents the one-dimensional vector of the first type of face ith image, and T

(1) (1) (1) xi(1)  xi1 xi2 . . . xi10304

(6.21)

(3) Training sample eigenvalue and eigenvector calculation Calculate the mean x¯1 of the first kind x¯1 

n1  m  1 x (1) n 1 × m i1 j1 i j

(6.22)

In the formula, xi(1) j represents the gray value of jth pixel of the ith samples of the first classes, and n 1 is the training sample number of the first types of faces. The mean x¯1 obtained by this method is the mean face of the first kind. After normalization of the first training samples, it can be expressed as ¯ i  1, 2, . . . , n vi(1)  xi(1) − x;

(6.23)

First classes of normalized vector v composed of training samples T

v1  v1(1) v2(1) . . . vn(1) At this point, there are first kinds of covariance matrices

(6.24)

6.2 Intelligent Switch Technology of Car Door

229

T 

n×n Q1  v1(1) v2(1) . . . vn(1) v1(1) v2(1) . . . vn(1) ; Q1 ∈ R

(6.25)

If the vehicle belongs to many people, that is, there are several owners, the same can be obtained second categories and more categories of facial features, namely the average face and covariance matrix expressions, such as Calculate the mean x¯l of the class l x¯l 

nl  m 1  x (l) n l × m i1 j1 i j

(6.26)

In the formula, xi(l) j denotes the gray value of the pixel j of the sample i of class l, and the mean x¯l is the average face of the class l. After standardizing the training sample of class l, it can be expressed as vi(l)  xi(l) − x¯l ; i  1, 2, . . . , n l

(6.27)

The normalized vector vl of a class l composed of training samples is T

vl  v1(l) v2(l) . . . vn(l)l

(6.28)

At this point, there is a class l covariance matrix T 

Ql  v1(l) v2(l) . . . vn(l)l v1(l) v2(l) . . . vn(l)l ; Ql ∈ R nl ×nl

(6.29)

Continue to compute the total mean x, ¯ when the number of samples per class is equal 1 x¯l c l1 c

x¯ 

(6.30)

The total mean x¯ is the mixed average face. Among them, c is the value of the test sample class. The interclass normalized vector v is obtained by the normalization of the mixed average face T  T

 v(l) v  v(1) v(2) . . . v(c)

(6.31)

Among them, v(l)  x¯l − x; ¯ l  1, 2, . . . , c, and then the interclass covariance matrix can be obtained T 

T c×c Q  v(1) v(2) . . . v(c) v(1) v(2) . . . v(c)  v v; Q ∈ R

(6.32)

230

6 Vehicle Auxiliary Intelligent Technology

Using the formula (6.31), the eigenvalue λll of Q and its eigenvector are obtained, and then the feature vectors are rearranged from large to small

T pl  λl1 λl2 λl3 . . .

(6.33)

Among them, λ1 ≥ λ2 ≥ λ3 ≥ . . ., and form a characteristic space array P with c×c ⎡

λ11  ⎢

λ ⎢ 12 P  p1 p2 . . . pc  ⎢ ⎢ λ13 ⎣ .. .

λ21 λ22 λ23 .. .

⎤ . . . λc1 . . . λc2 ⎥ ⎥ ⎥ . . . λc3 ⎥ ⎦ . . .. . .

(6.34)

And P T P  I (4) The training samples are transformed into feature space by the linear transformation Due to the larger eigenvalues, the corresponding eigenvectors contain more facial feature information. Therefore, it is possible to approximate the main information of the face image by selecting the vector space of the eigenvectors corresponding to the larger eigenvalues of the previous sl . For the n l images in the image library, T

(l) (l) (l) xi(l)  xi1 (i  1, 2, . . . , n l ) can be projected to the feature space, xi2 . . . xim T

(l) (l) (l) is obtained. and the projection vector i(l)  ωi1 ωi2 . . . ωim A new normalized vector vˆ l is constructed by selecting the normalized values corT

responding to the larger eigenvalues of the preceding sl from vl  v1(l) v2(l) . . . vn(l)l T

vˆ l  v1(l) v2(l) . . . vs(l) l And the corresponding feature space Pˆ with n l × c ⎡ ⎤ λ11 λ21 . . . λc1 ⎢ ⎥ ⎢ λ12 λ22 . . . λc2 ⎥ ⎥ ˆP  ⎢ ⎢ . . . ⎥ ⎢ .. .. . . ... ⎥ ⎣ ⎦ λ1sl λ2sl . . . λcsl

(6.35)

(6.36)

So there is 

(l)  1(l) 2(l) . . . (l)  Pˆ T vˆ l nl

(6.37)

6.2 Intelligent Switch Technology of Car Door

231

Fig. 6.15 Test samples 1

Fig. 6.16 Test samples 2

Therefore, (l)  Pˆ T vˆ l can be used to represent the face of class l. (5) Import test samples Suppose that the face sample number for a test is n, then the test set is the matrix with m × n. Suppose a car owner approaches the door, and the image of his face is collected, as shown in Fig. 6.15. (6) Calculate the image feature vector of the sample to be tested According to the above-mentioned 3rd and 4th steps, the eigenvalues and eigenvectors of the test sample image are computed. The test sample is projected into the characteristic subspace represented by formula (6.37). At the same time, the face image is projected to the feature subspace, which will correspond to a point in the subspace. (7) Face recognition Each test image and the training image projected to the subspace are compared one by one to determine the category of the sample to be identified. For example, the nearest neighbor distance classification function is used for recognition     (6.38) G , (l)  min  − (l)  l

The  here represents the feature subspace of the test sample, and the formula (6.38) can confirm whether the sample to be tested belongs to the training sample. For example, the test samples in Fig. 6.15 are clearly recognized as the trained first types of faces (see Fig. 6.14), that is to say, the test is the owner (or one of the owners). In other words, if close to the car door, the test process does not belong to any class of training samples, such as the test sample shown in Fig. 6.16, which means that the person close to the door is not the owner of the vehicle.

232

6 Vehicle Auxiliary Intelligent Technology

To complete the face test and confirm that the person belongs to the owner, the system will further fingerprint identification. Otherwise, the system work interruption, the door is always in the closed state.

6.2.2.2

Fingerprint Identification Algorithm

Fingerprint identification algorithms include: fingerprint registration (Register), identification (Identify) and verification (Verify). Fingerprint registration refers to the collection of user fingerprint, feature extraction and template is saved to the database. Identification refers to the acquisition of unknown user fingerprint, extraction of feature information, and the template stored in the database to match, to obtain user identity. Verification refers to the acquisition of a known user fingerprint, extraction of feature information, and the user-specified database of a fingerprint template matching, to determine whether the two templates belong to the same finger of the fingerprint. 1. Fingerprint stitching Each time the fingerprint is collected, a sliding fingerprint sequence can be obtained from the sensor, and there is a lot of repeat areas between the fragments, and a complete fingerprint image is obtained by splicing the fingerprint sequence. Because there is no rotation transformation and scaling transformation between adjacent fingerprint segments, only the overlap of some region images exists. Therefore, it is important to judge the shift of the obtained fragments relative to the previous one. Shift algorithm can be directly carried out in the time domain, that is, statistics of the gray level changes of each row of images in the horizontal direction, and then determines the shift through this statistical information. In order to express convenience, let a be the previous fragment, and b is the latter fragment. (1) Median filtering for a and b respectively, in order to weaken the influence of noise on stitching. (2) Calculate the mean value of a and b respectively and make the second value. (3) Each column image is counted from left to right to calculate the number of pixels at the gray level where the jump points appear. (4) b is sliding on a, and each sliding column calculates the similarity S of b and a overlap region, and obtains a sequence of S. When the maximum value of S occurs, the corresponding column number m is the translation of b relative to a. (5) After determining the number of shift  columns m, based on the number of columns in the overlap region, the m2 columns are pushed forward as the last column of the a fragment from the last column of the overlapping region of the a fragment. From the front column of the overlap region of the b segment, the  m columns are pushed back as the new front column of the b fragment. Then, 2 the new last column of the a fragment is spliced with the new front column of the b fragment. And so on, finally, the whole sliding fingerprint sequence is

6.2 Intelligent Switch Technology of Car Door

233

spliced. Among them, · and · respectively take the integer from the down and up to the result of the calculation. Through the above stitching algorithm operation, the pixel shift caused by the sliding of the initial fingerprint image can be well compensated. The fingerprint capture image and its stitching effect are shown in Fig. 6.17. Among them, Fig. 6.17a is the initial sliding fingerprint image obtained by the semiconductor fingerprint sensor. Figure 6.17b is a mosaic image of the initial sliding fingerprint image which is realized by the above stitching algorithm. Because of the sliding fingerprint image splicing operation is performed in the time domain, without the use of image transformation, which did not adopt the method of frequency domain conversion processing, so a small amount of calculation, the system operation occupies fewer resources, so it can greatly shorten the operation time required for splicing. If AT77C104A FingerChip sliding fingerprint sensor is used in 200 MHz working frequency, the time needed for a moving fingerprint image to be spliced is less than 10 ms. 2. Fingerprint feature extraction The feature extraction of fingerprint image includes the following steps: (1) Image enhancement Because the actual collected fingerprint image may contain the background region without ridge information, and at the same time, there are various noises in the image region containing ridge information, which leads to the adhesion or fracture of the ridges. Therefore, it is necessary to enhance the fingerprint image, cut background area, increase the contrast between ridges and valleys, weaken noise pollution, separating the ridges of adhesions, connecting fractured ridges. As shown in Fig. 6.18a is the image of a sliding fingerprint collected by the sensor through the stitching. Figure 6.18b is the result of enhancement and filtering of the fingerprint image. (2) Two-valued The enhanced fingerprint image is two-valued and refined, as shown in Fig. 6.19. Among them, Fig. 6.19a is a two-valued image for the enhanced fingerprint image. Figure 6.19b is a further refinement result of the two-valued image.

Fig. 6.17 Fingerprint acquisition and stitching effect legend

234

6 Vehicle Auxiliary Intelligent Technology

Fig. 6.18 Fingerprint enhancement effect legend

Fig. 6.19 Two-valued legend of fingerprint

Fig. 6.20 Sketch of fingerprint feature points

(3) Fingerprint feature extraction Looking for two kinds of fingerprint feature points from the refined two-valued images. That are, the bifurcation points and the end point, and calculate the direction angle of the feature point. Finally, the location and orientation information of feature points are used as fingerprint features to be stored in fingerprint feature space database. The characteristic points obtained from the two-valued and refined fingerprint images of Fig. 6.19b are shown in Fig. 6.20. 3. Matching of fingerprint feature points The matching algorithm of fingerprint feature points includes two steps: ridge calibration phase and feature matching stage.

6.2 Intelligent Switch Technology of Car Door

235

(1) Calibration stage of ridgeline In this stage, two fingerprint images are located, and the influence of translation and rotation is eliminated. The origin of the corresponding feature points is calculated as the coordinate of the feature points. (2) Feature matching stage This stage is based on the calibration of the ridge line, according to the corresponding feature point pairs. Where the feature points set of fingerprint samples in feature space and the sampled fingerprint feature points set are transformed to polar coordinate plane, according to the polar coordinates and the direction information of the position of the feature points, the matching of the pair of feature points is searched and the matching point pairs are counted. When the number of pairs of matching points reaches a given threshold, the match can be considered successful, terminate matching at the same time, it can be concluded that the currently tested fingerprint belongs to the fingerprint sample which is successfully matched in the feature space. If the matching fails, it is considered that the tested fingerprint is not a fingerprint sample in the feature space.

6.3 Intelligent Control of Vehicle Windscreen Wiper Vehicle wiper is usually operated by hand. After the windscreen wiper, the intelligent control system was installed, it can according to whether the rain and rainfall intelligent size to automatically achieve the wiper movement control. Therefore, there is no need for manual wiper when driving in rainy days, and the driver does not need to adjust the wiper settings. The system can make the wiper movement achieve the optimal state. When driving on a wet road, if the water splashes on the windshield, it naturally does not need the driver to do it, and the wiper will automatically start and stop. Therefore, the driver can concentrate on driving, greatly improving the comfort and safety of car driving in rainy days.

6.3.1 Rain-Sensing Principle Infrared sensors can be used in rainy days. The infrared light has little influence on the environment and is easy to detect. Where a pair of semiconductor light emitting elements and a photosensitive element are matched, light signals emitted from light emitting elements, if raindrops are encountered in the road, light scattering and light intensity are weakened, and the weather information of rain weather can be picked up by detecting the attenuation signal of light intensity. However, when infrared sensing rain information, the infrared sensor must be paired, the installation of the car is more complex, and the receiver tube once the object is blocked, it is easy to lose efficacy.

236

6 Vehicle Auxiliary Intelligent Technology

Fig. 6.21 Images that appear through the front windshield when light rain occurs

This paper introduces a method of using the compensatory fuzzy neural network to identify rainy days. Which can better solve the problem of rainy days identification, and the reliability is higher.

6.3.1.1

Rain Image Features

(1) Due to the erosion of rainwater, the asphalt road surface will become darker, and the road signs will become whiter. When the rain is small, the road sign is not flooded by rain, and the contrast between the landmarks increases. These images will appear the boundary-bending phenomenon after the car front windshield is transmitted into the cab. As shown in Fig. 6.21. (2) When rainfall is large, a runoff will occur, road signs and roads are inundated with rainwater, the landmarks and roads in the captured image become blurred. Because of the ripples produced by raindrops, the waves will appear in the image, and show a partial reflection. As shown in Fig. 6.22. (3) When the big (storm) rain, the front windshield of the whole car becomes a water curtain, in addition to the light refracted by the water curtain, hardly see the scene ahead. As shown in Fig. 6.23.

6.3.1.2

Compensation Fuzzy Neural Network Structure

Compensation fuzzy neural network is a hybrid system combining fuzzy compensation logic and neural network, which is composed of fuzzy neuron oriented to control and decision making. The fuzzy neuron in the compensation fuzzy neural network is defined as performing the fuzzy operation, fuzzy inference, compensating fuzzy operation and defuzzification operation.

6.3 Intelligent Control of Vehicle Windscreen Wiper

237

Fig. 6.22 Front windshield images during moderate rainfall

Fig. 6.23 Images that appear through the front windshield during heavy rain

Fuzzy reasoning stores knowledge in the rule set, the neural network stores knowledge in the weight coefficient. The compensation fuzzy neural network is introduced to compensate the fuzzy neurons, so that the network can be trained from the initial fuzzy rules or the fuzzy rules defined by the initial errors, connect in series, that is, the output of one side becomes the input of the other, this kind of situation can be regarded as the two-part inference or the pretreatment part of the input signal in the series, which makes the network fault tolerance higher and the system more stable. Meanwhile, dynamic and global operations are used to compensate the fuzzy operations in the fuzzy neural network. Moreover, the fuzzy compensation operation is dynamically optimized in the neural network learning algorithm, which makes the network more adaptive and more optimized. The compensation fuzzy neural network can not only adjust the fuzzy membership function of input and output, but also optimize the fuzzy reasoning dynamically with the help of the compensation logic algorithm and accelerate the training speed. The

238

6 Vehicle Auxiliary Intelligent Technology

Fuzzy inference layer

Defuzzification layer

Input layer

Fuzzy layer

Compensating operation layer

Fig. 6.24 Compensatory fuzzy neural structure diagram

compensation fuzzy neural network based on the above features is used to identify the compensatory fuzzy neural structure in rainy days, as shown in Fig. 6.24. Compensation fuzzy neural network has 5 layers of structure, including input layer, fuzzification layer, fuzzy inference layer, compensation operation layer and defuzzification layer. Layers and layers are constructed according to the linguistic variables of fuzzy logic system, fuzzy IF-THEN rules, worst/best operations, fuzzy reasoning methods and deblurring functions. (1) Input layer The layer has n nodes  directly connected with the input vector x, and the input value  x  x1 x2 . . . xn is transmitted to the next layer. Here, the input layer contains 3 neurons, and the input vector is     x  x1 x2 x3  R G B

(6.39)

In the formula, R, G and B represent the color component values of image pixels. (2) Fuzzy layer Because each fuzzy input set contains 3 Fuzzy quantities, the layer contains 3 × 3  9 neurons. The 9 neurons are divided into 3 groups, the input of three nodes in each j group is xi (i  1, 2, 3), and the output is the membership function μi (xi ) of the j fuzzy subset of each input quantity which belongs to the output value. In other words, each node of the 9 neurons represents a linguistic variable value, whose function is to compute the membership functions of each input vector belonging to the fuzzy set of linguistic variable values. The R, G and B are blurred into 3 sets of small (S), medium (M) and large (L). Gauss function expresses the fuzzy membership degree, that is

6.3 Intelligent Control of Vehicle Windscreen Wiper Table 6.2 Fuzzy rule table Rule IF THEN 1 2 3 4 5 6 7 8 9

LLL LLM LLS LML LMM LMS LSL LSM LSS

L L ML L ML M LM LM M

239

Rule

IF

THEN

Rule

IF

THEN

10 11 12 13 14 15 16 17 18

MLL MLM MLS MML MMM MMS MSL MSM MSS

L ML M ML M SM M SM S

19 20 21 22 23 24 25 26 27

SLL SLM SLS SML SMM SMS SSL SSM SSS

ML M SM M SM SM SM S S

⎧

   ⎪ xi −120 2 S ⎪ μ  1 − exp − (x ) ⎪ i A ⎪ 25 i ⎪ ⎪ ⎨

   xi −130 2 μM Ai (x i )  1 − exp − 10 ⎪ ⎪ ⎪

 ⎪ 2  ⎪ ⎪ ⎩ μ LAi (xi )  1 − exp − xi −140 25

(6.40)

In the formula, i  1, 2, 3. Ai is a fuzzy set on the domain U . j  1, 2, . . . , m  1, 2, 3  S, M, L. (3) Fuzzy inference (rule) layer There are 27 neurons in the layer, each node represents a fuzzy rule, whose function is to match fuzzy rules, and calculate the applicability of each rule. The fuzzy rules are shown in Table 6.2. The 27 fuzzy IF-THEN rules in Table 6.2 are expressed as follows j

IF x1 is Ai and . . . xn is Anj THEN y is B j

(6.41)

j

Among them, Ai is the fuzzy set on the domain U , j  1, 2, . . . , m  1, 2, 3  S, M, L, B j is the fuzzy set on the domain V , xi and y are linguistic variables, i  1, 2, . . . , n  1, 2, 3. The output is blurred into small (S), small-medium (SM), medium (M), mediumlarge (ML), large (L). (4) Compensation operation layer There are 10 neurons in thelayer, which  compensate   the fuzzy rules. According to Table 6.2, for the input x  x1 x2 x3  R G B , the domain is U  U L ×U M × U S , and for the fuzzy input subset A in the domain U , according to the k fuzzy rule, an output fuzzy subset B  can be generated in the output domain V . Then, the membership function of the fuzzy set B  on the domain V is derived from the fuzzy inference rules, that is

240

6 Vehicle Auxiliary Intelligent Technology



j μ B  (y)  sup μ A j ×A j ×···×Anj (x, y) · μ A (x) 1

X ∈U

2

(6.42)

When fuzzy reasoning uses maximal algebraic product composition operations, the formula (6.42) can be expressed as 

j (6.43) μ B  (y)  sup μ A j ×A j ×···×Anj (x)μ B j (y)μ A (x) 1

x∈U

2

Positive operation of fuzzy fitness μj 

n 

μ A j (xi )

(6.44)

i

i1

And negative arithmetic ν  j

n 

! γn μ A j (xi )

(6.45)

i

i1

Then, the compensation operation of μ A j ×A j ×···×Anj (x) is as follows 1

2

n   1−γ j μ A j ×A j ×···×Anj (x)  μ j ν  μ A j (xi ) 1

2

!1−γ + γn

i

(6.46)

i1

Therefore, the formula (6.42) can be expressed as ⎡ !1−γ + γn ⎤ n  j ⎦ μ A j (xi ) μ B  (y)  sup⎣μ B j (y) · μ A (x) i

x∈U

(6.47)

i1

Due to the use of single-valued fuzzification, namely μ A (x)  1,μ B k (y)  1, there is a compensation result j μ B  (y)



n 

!1−γ + γn μ A j (xi ) i

(6.48)

i1

(5) Defuzzification layer There are only 1 neurons in this layer, and defuzzification processes the input, that is, the output of y is as follows

6.3 Intelligent Control of Vehicle Windscreen Wiper

#

m "

b δ j

j1

y

m "

j

# δj

j1

n $

j μ Ai (xi )

i1 n $

i1

241

j μ Ai (xi )

%1−γ + γn

%1−γ + γn

(6.49)

In the formula, b and δ are the center and width of the output membership function, γ is the degree of compensation, and generally γ  0.3.

6.3.1.3

Compensation Fuzzy Neural Network Learning Method

    For 3D input data x  x1 x2 x3  R G B and one-dimensional output data y, the learning method of compensating fuzzy neural network is to adjust the center and width of input and output membership function of the fuzzy neural system to meet the logical relationship between input and output. It is assumed that the error evaluation function E is E

2 1 y − y∗ 2

(6.50)

In the formula, y ∗ is the ideal output. The learning process of compensating fuzzy neural networks is as follows: 1. The center, width, and weight of the input membership function are trained (1) Training input membership function center j

j

bi (t + 1)  bi (t) − ηβ

∂E j

∂bi

 j j + ξ bi (t) − bi (t − 1)

(6.51)

j

In the form, bi (t) is the center of the ith input at the moment t. η as learning rate. ξ is the coefficient of inertia, ξ ∈ [0, 1]. m $

&   γ ' j1 ∗  2 y − y − y) 1 − γ + (ω i j n ∂bi ∂E

 j j j j μ Ai xi + θi μ Ai (t − 1) − bi & '2 j ρi j

(6.52)

In the formula, ωi is the weight of ith input. θi is the connection weight of jth j fuzzy subsets and the recursive units of ith input. ρi is the membership width of jth fuzzy subset and ith input.

242

6 Vehicle Auxiliary Intelligent Technology

(2) Training input membership function width j

j

ρi (t + 1)  ρi (t) − ηβ

∂E j

∂ρi



j j + ξ ρi (t) − ρi (t − 1)

m $

&   γ ' j1 ∗ 1 − γ +  2 y − y − y) (ω i j n ∂ρi ∂E

# j

μ Ai

n "

i1

j

j

(6.53)

j

xi + θi μ Ai (t − 1) − bi &

j

ρi

2 %

'3 (6.54)

(3) Training input recursive unit connection weight j

j

θi (t + 1)  θi (t) − ηβ

∂E ∂θi

j

 j j + ξ θi (t) − θi (t − 1)

(6.55)

&   γ' ∗  −2 y − y − y) 1 − γ + (ω i j n ∂θi

  m $ j j j j j μ Ai xi + θi μ Ai (t − 1) − bi μ Ai (t − 1) ∂E

j1

& '2 j θi

(6.56)

(4) Training input weight j

j

ωi (t + 1)  ωi (t) − ηβ

∂E j

∂ωi

 j j + ξ ωi (t) − ωi (t − 1)

m & γ ' j  (ωi − y) 1 − γ + μ (xi ) j n j1 Ai ∂ωi

∂E

(6.57)

(6.58)

2. The center and width of the training output membership function (1) The center of the training output membership function ∂ E b (t + 1)  b (t) − η j ∂b t j

j

(6.59)

(2) The width of the training output membership function δ j (t + 1)  δ j (t) − η

∂ E ∂δ j t

(6.60)

6.3 Intelligent Control of Vehicle Windscreen Wiper Pinhole camera

Signal processor

243 Controller

Wiper

Fig. 6.25 Block diagram of wiper control system

6.3.2 Practical Technology of Wiper Intelligent Control 6.3.2.1

System Composition Module

The vehicle wiper intelligent control system consists of the image sensor, signal processor, controller and wiper module, as shown in Fig. 6.25. The image sensor is a pinhole camera mounted in the driver’s cab toward the front windshield plate. The hardware of the signal processor is a vehicle-mounted embedded system, and the key is the running program. The core program of the vehicle wiper intelligent control system is to compensate fuzzy neural network algorithm. Usually, the signal processor is responsible for the processing, analysis, and recognition of the sensing signal. The compensation fuzzy neural network algorithm is based on the identification of rain image, the recognition results in binary code output control command to the controller, the controller controls the wiper mode. The controller is actually a decoder that bears the explanation of the received control instructions. Of course, there are a lot of dedicated chips specially designed for the wiper, which can assume the function of this controller, that is, the controller can be directly made up of a special wiper control chip to form wiper control circuit. As far as the wiper control is concerned, the working state of the wiper can be divided into several ways: Fast scraping, Normal scraping, Periodic scraping, Intermittent scraping, etc. At the same time, in different scraping state, but also to control the frequency of its wiper action. Take the special wiper controller chip SX5125 as an example, the application circuit is shown in Fig. 6.26. In the SX5125 chip application circuit, there are 5 control ports and a homing feedback port (see Fig. 6.26). Among them, the GP is the scraping frequency control terminal, ZG is the normal scraping control port, KG is a fast wiper control port, XG is the scraping control port, GW is the homing signal port, JG is the intermittent scraping control port, RW1 is a sliding resistor, which acts as an auxiliary regulator of clearance time interval, the greater the resistance value, the smaller the scraping gap time, can be set as predetermined resistance, of course, the gap scraping time interval is mainly modulated by the output instruction of the signal processor. The relationship between the above working states and the control ports is shown in Table 6.3. In Fig. 6.26, the voltage signal output by the OUT1 or OUT2 determines the turnon or turn off of the IGBT through the power electronic switching device. Therefore, the H bridge circuit made up of IGBT can make the servo DC motor rotate in a positive/reverse direction according to the specific rules, and then drive the wiper to move back and forth. At the same time, voltage signal OUT1 or OUT2 output is duty

244

6 Vehicle Auxiliary Intelligent Technology

Fig. 6.26 Wiper controller application circuit Table 6.3 the corresponding table of control port/output truth value JG KG ZG XG GW

OUT1/OUT2

0

1

0

0

0

Fast scraping

0

0

1

0

0

Normal scraping

0

0

0

1

0

Periodic scraping

1

0

0

0

0

Intermittent scraping

cycle adjustable digital pulse. Therefore, the equivalent drive current of the motor is proportional to the duty cycle of the OUT1 or OUT2 output voltage. Taking the normal scraping as an example, the corresponding relationship between the output waveform of GS, ZG, GW, OUT1, OUT2 and its time is shown in Fig. 6.27. In Fig. 6.27: (1) The period TZ G of ZG is variable, that is, it is adjusted according to the size of rainfall. The larger the rainfall, the shorter the cycle. (2) GP follows the periodic change of ZG, which means, the frequency FG P of GP is related to TZ G FG P 

k1 TZ G

(6.61)

In the formula, k1 is the proportional coefficient. (3) The positive pulse width of GW is the intermittent time τ of wiper, and the size of τ is related to FG P (or TZ G ), the higher the FG P (or the smaller the TZ G ), the smaller the τ , which can be expressed as

6.3 Intelligent Control of Vehicle Windscreen Wiper

S1

S2

S3

S4 M

245

t

Fig. 6.27 Waveform curves of GP, ZG, GW, OUT1 and OUT2

τ  k 2 TZ G

(6.62)

In the formula, k2 is the proportional coefficient. (4) The output of OUT1 and OUT2 is the output waveform of GP in the positive signal of ZG gate, and the logical relation is as follows OU T 1 or OU T 2  G P ∧ Z G

(6.63)

(5) The current flowing through S1, S2 or S3, S4 depends on the duty cycle of OUT1 or OUT2, when the duty cycle is large, the equivalent current flowing through the power electronic switching device (IGBT) is large. Under the action of S1, S2 conduction and cutoff, at the same time, under the action of S3, S4 cutoff and conduction, the driving current of the motor M are shown in the last curve of Fig. 6.27.

246

6.3.2.2

6 Vehicle Auxiliary Intelligent Technology

System Master Control Flow

The process of intelligent control of wiper is as follows: 1. Image acquisition Real-time acquisition of windshield images using a pinhole camera mounted in the driver’s cab toward the front windshield. 2. Rainy days recognition operation The compensation fuzzy neural network algorithm is used to identify the current collected windshield image, judge whether the current rain, and determine the size of the rainfall. (1) Constructing compensation fuzzy neural network structure, it includes input layer, fuzzification layer, fuzzy inference layer, compensation operation layer and defuzzification layer. (2) The R, G and B color of the image are used as input values, and an   components input vector x  x1 x2 x3 is set up to transmit to the next layer. (3) According to the small (S), middle (M), large (L) 3 sets of rainfall, each input is fuzzification, the fuzzy membership degree is used as the distribution law of membership degree by Gauss function, that is ⎧

   ⎪ xi −120 2 S ⎪ μ  1 − exp − (x ) ⎪ i ⎪ Ai 25 ⎪ ⎪ ⎨

   xi −130 2 μM Ai (x i )  1 − exp − 10 ⎪ ⎪ ⎪

 ⎪ L 2  ⎪ ⎪ ⎩ μ Ai (xi )  1 − exp − xi −140 25 In the formula, i  1, 2, 3 Ai is the fuzzy set of R, G and B color components on the image pixel domain U , j  1, 2, . . . , m  1, 2, 3  S, M, L. (4) On the 27 neurons of the fuzzy inference (rule) layer, the fuzzy rules corresponding to each node are determined according to the fuzzy rules, such as Table 6.2. (5) The fuzzy rules of the 10 neurons on the compensation operation layer are compensated according to formula (6.42)–(6.48). (6) The result of compensation operation is defuzzification, and the output y is obtained.

6.3 Intelligent Control of Vehicle Windscreen Wiper

247

Image acquisition

Fuzzy neural network

Control command output

Control instruction decoding

Compensation fuzzy neural network algorithm

Fuzzy inference rule

OUT1/OUT2 output

H bridge circuit of IGBT

Servo DC motor

Fig. 6.28 Intelligent control flow chart of wiper

3. Control command output The identification result y is converted to the wiper controller according to the logical relation of Table 6.3 and the corresponding wiper frequency. 4. Drive wiper motor Wiper controller by the received control instructions to control Fast scraping, Normal scraping, Periodic scraping, Intermittent scraping, etc. working methods. The wiper intelligent control flowchart is shown in Fig. 6.28.

6.4 Automatic Steering Technology of Automobile Headlamp When the car runs in the night, when it meets the curve and the specific section; the driver cannot see the area outside the headlight beam directly. Which will directly affect the driving safety, therefore, the ideal headlamp should be able to automatically follow the steering wheel steering lighting, to enable the driver to see the road in front of the bend in front of the road conditions.

248

6 Vehicle Auxiliary Intelligent Technology

Fig. 6.29 Light beam of automobile headlamp on curved road

6.4.1 Beam Distribution of an Automobile Headlamp at the Curved Road In general, the headlamp of an ordinary car is fixed when it is factory, and its optical axis is basically perpendicular to the axle. Once the position of the condensing lens (or lamp holder) is adjusted manually, the divergence angle of the headlamp is determined, the visual range of the driver at night can only be limited to the bright area ahead of the vehicle, as shown in Fig. 6.29. The fixed front angle lighting method of the vehicle headlamp can cause the driver not to see the road condition of the turning lane that will enter, especially, the vehicles running on the complex mountain roads at night often suffer traffic accidents due to the misjudgment of drivers’ road conditions. It can be seen that if the headlamp can follow the direction of the vehicle to turn the lighting direction accurately, it will inevitably bring great convenience to the driver.

6.4.2 Automatic Steering Technology of Automobile Headlamp Two ways can be used for the automatic steering of the headlamp: Headlamp for automatic following vehicle steering and Side assist lighting.

6.4 Automatic Steering Technology of Automobile Headlamp

6.4.2.1

249

General Headlamp for Car

The general headlamp of an automobile includes four parts: reflector, astigmatic glass, bulb and light adjusting mechanism. The ordinary headlamp also includes three types: enclosed, semi-enclosed and combined headlamp. (1) Enclosed headlamp The automobile closed headlamp includes two different closed vacuum wicks, fixing rings, adjusting seats, fixing frames, adjusting springs, adjusting screws and screw seats. This is a typical auto four lamp headlamp system. Among them, I# vacuum wick only has a group of far beam filament, there are two groups of filaments in E# vacuum wick, that is, far beam and near the light. In the automotive assembly, two E# vacuum wicks must be installed at the outermost side of the front end of the car. The closed vacuum wick mirrors are mostly paraboloid formed by parabola rotation. In the process of the mirror production, the aluminum coating is evaporated on the inner surface with the vacuum coating process. The brightness of the reflected light is increased by more than 6000 times than the brightness of the filament itself. The bulb filament is made up of winding tungsten wires, and the filament of the far beam is mounted on the focus of the paraboloid of the reflector. The filament of near light is mounted above the focus and slightly left (relative to the vehicle passing on the right). When the far beam filament is illuminated, the light emitted by the filament is reflected by the reflector, and the light is parallel to the distance along the optical axis, as shown in Fig. 6.30a. The near-light emitted by the filament is reflected diagonally below the reflector when the light near the filament is turned on, as shown in Fig. 6.30b. The reflector reflects the total luminous flux emitted by the filament, and the useful luminous flux under the inclined direction is much larger than the luminous flux above the oblique direction, thereby reducing the dazzling effect of the driver of the opposite vehicle. Because the near-light filament near the left slightly, but also make the effective downward beam to the right of an angle. The light beam headlight of the automobile is formed through the reflector is difficult to meet the traffic regulations of headlamp requirements, also need the lens

Fig. 6.30 Schematic diagram of far and near light beam formation of headlamp

250

6 Vehicle Auxiliary Intelligent Technology

on the beam with light changes or widened or narrowed, so to form the required lighting in front of the car, this function depends on the headlamp light glass to complete. Headlamp astigmatic glass is composed of many concave and convex small prisms. It can refract and scatter the reflected light from the reflector so as to meet the requirements of the headlamp, at the same time, a part of the light is diffused to both sides to widen the illumination range of the headlamp in the horizontal direction, and the desired light distribution effect is obtained. (2) Semi-closed headlamp (a) Small size semi-closed headlamp The light distribution mirror and reflector of small size semi-closed bonding forms headlamp used in automobiles or rolling. The structure has a semi-removable metal glass component or a bonding metal glass component. The mirror is made of steel plate stamping, and modern technology adopts thermosetting plastics injection molding. The light bulb of the semi-closed headlamp is usually loaded or replaced from the rear. Because of its simple structure, low cost and easy maintenance, the headlamp is widely used in trucks and agricultural vehicles. (b) Large size semi-closed headlamp Semi-enclosed headlamp for an automobile, besides circular headlamp, there is rectangular semi-closed headlamps. The light distribution mirror and the reflector of the headlamp are individually detachable because the headlamp size is large. If the light distribution mirror and the reflector are also fixed connection, when the light is adjusted in horizontal direction, the side of the astigmatic glass will produce larger front and back linear displacement up to 15–20 mm, this will cause the front of the vehicle is extremely unsightly, sometimes stuck on the edge of the outer decorative ring, so that the lights cannot be adjusted. The same problem occurs in the vertical adjustment of the light. Therefore, the headlamp is often designed as a reflector, which is adjustable. When the light needs to be adjusted, the reflector can only be adjusted, while the light distribution mirror is not moving. (c) Combined headlamp In recent years, the combination headlamps are often used in cars all over the world. It is often designed according to the aesthetic point of view of vehicle styling, and it can be coordinated with the vehicle shape, and can reduce the drag coefficient of the vehicle. This headlamp is also known as a special headlamp. In addition to the function of the ordinary headlight near the light and far beam function, it also has the function of parking lights, steering lights and so on. The headlamp is an ordinary headlamp which is fixed in the direction of the beam emission, and the steering headlamp does not need to be changed in the optical structure.

6.4 Automatic Steering Technology of Automobile Headlamp

6.4.2.2

251

Vehicle Automatic Turning Headlamp

The vehicle steering headlamp, a headlamp that can change the beam’s adaptability when following the vehicle, it’s called automatic steering headlamp. The basic structure of the automatic steering headlamp is similar to that of the combined headlamp (see Fig. 6.31), but a drive mechanism already drives its base support handle (9 of Fig. 6.31), the support handle can rotate with the vehicle to make the adaptive beam rotation. The relation equation between the optical axis angle of the automatic steering headlamp and the steering of the vehicle can be expressed as ( V ·φ (6.64) α  0.5k 15 In the formula, the φ is the steering wheel angle (radian), k is the steering angle of the steering wheel and the steering ratio of the vehicle, V is the vehicle’s current speed (km/h), α is an automatic steering headlamp optical axis angle, the rotation angle of the handle (in radian). The changes of φ, α and vehicle steering angle are based on the straight vehicle and the direct light of the headlamp. Take the left turn of the vehicle as an example, a vehicle with automatic steering headlights. When it turns to the left, the headlamp can track the turning of the vehicle synchronously and deflect the emission beam of the headlamp appropriately (see Fig. 6.32), it allows the driver to observe the left lane traffic very clearly. The dashed lines in the graph represent the light emitting region of the headlight without automatic steering.

Fig. 6.31 Headlamp structure diagram, (1 Lamp housing, 2 Lamp ring, 3 Reflector, 4 Light bulb, 5 Astigmatic glass, 6 Bulb holde, 7 Shell seat, 8 Cable, 9 Rotating handle)

252

6 Vehicle Auxiliary Intelligent Technology

Fig. 6.32 Schematic diagram of steering effect of automatic steering headlamp Fig. 6.33 Beam distribution of headlamp and side assist lamp

6.4.2.3

Auxiliary Lighting Lamp for Side of Vehicle

In order to ensure that the driver can see the road in real time, the turning assistant lighting device can also be used when the vehicle turns at night. In this device, the position of the turn auxiliary light and the headlamp is fixed, but the real-time control of the turning time is used to achieve the function of turn auxiliary lighting. The structure of the headlamp and side assist lamp is shown in Fig. 6.33. This kind of car in addition to conventional far, near the light, is also equipped with lateral auxiliary lighting. The vehicle controller controls the open/close rule of side auxiliary lighting according to the turning state of the vehicle, the general mathematical model is as follows

6.4 Automatic Steering Technology of Automobile Headlamp

⎧ ) * ⎪ kφ ⎪ ⎨ 1 ; if α  0.1 10 × β ≥ 1.25 I  * ) ⎪ kφ ⎪ ⎩ 0; if α  0.1 10 × β < 1.25

253

(6.65)

In the mode, I is the side auxiliary lighting switch signal, I  1 represents an opening, and I  0 represents closing. φ is the angle of rotation of the steering wheel (radian). k is the transmission coefficient. β is the angle (radian) of the optical axis corresponding to the side edge of the main beam of the headlamp. α is a parameter variable to be determined. · represents the operator of down integers. The formula (6.35) means that when the vehicle turns over a certain angle, namely α ≥ 1.25, the onboard signal processor will automatically transmit the control command I  1 and open the side auxiliary lights in real time so that the driver can clearly observe the traffic condition which will be transferred into the lane. Otherwise, the side auxiliary lights are in the closed state, that is I  0.

6.5 Automatic Response Technique for Parking Location With the emergence of large parking lots and underground parking lots, car owners often have to spend a lot of time looking for cars because they forget the parking position (or without memory), which can cause inconvenience and even affect the driving emotion afterward. For example, such incidents have occurred at underground multistory parking lots at Beijing Capital International Airport. And for example, in the large shopping mall parking lot, the owner out of the store, because the car did not pay attention to the parking position in advance, the result spent a considerable amount of time, from a large range of “one by one investigation” thing, is not uncommon. There are two kinds of parking position automatic response technology: GPS positioning method and positioning method based on mobile operating base station. Vehicle position automatic response technology allows owners to find their own car parking position, very worry, convenient.

6.5.1 Automatic Response Technique for Ground Parking Position Ground parking location response technology is based on GPS positioning technology. The technology can receive GPS signals directly, so it can realize the automatic response of ground parking position by using the vehicle GPS and GPRS joint system.

254

6 Vehicle Auxiliary Intelligent Technology

GPRS

GPRS

Vehicle mounted GPS

GPRS Fig. 6.34 Schematic diagram of GPRS mobile communication system

6.5.1.1

System Basic Constitution

The system structure of “ground parking position automatic response” is implemented by vehicle GPS, as shown in Fig. 6.34. The system carries on the localization data transmission by the vehicle GPS through the RS232 interface and the vehiclemounted GPRS terminal, vehicular GPRS terminals communicate with GPRS mobile phone through GPRS backbone network.

6.5.1.2

Principle of Automatic Response of Ground Parking Position

In advance, the onboard GPRS terminal must have access to the IP address and its corresponding terminal number authorization through the mobile communication operator. When the owner returns to the parking lot, but forgets his parking space and needs to find the vehicle position, the GPRS mobile phone can be used to send the agreed SMS to the vehicle GPRS terminal, and the system is initialized by network dialing, PPP negotiation, TCP connection and other operations. When the network connection between the vehicle terminal and the server is established, the data exchange between the vehicle terminal and the server can be realized through the communication protocol between the user-defined vehicle terminal and the server. In this process, the system will first determine whether the current system allows the terminal to upload GPS data, and the corresponding operation. At the same time, it will check whether the control commands are received (including commands issued via the network, or sent by SMS, or by remote control commands). If the control command is received, the legitimacy of the command is judged and dealt with accordingly. At the same time, the system will regularly test the quality of the network, if the network test is normal, it returns to continue to judge the control command and process the current state of the system. The onboard GPRS terminal immediately sends the real-time GPS location information (geographic coordinate data) to the connected GPRS mobile phone. The GPRS mobile phone can learn the location information of the vehicle to be found from the received SMS. In this process, if the network test is abnormal, a limited attempt is made. If the limited attempts are unsuccessful, and actively disconnect the network link, and GPRS

6.5 Automatic Response Technique for Parking Location

255

reset and re-initialization operation: repeat the main program dial, PPP negotiation, TCP connection and network data interaction work. The working procedure of the vehicle GPRS terminal control program is as follows: (1) (2) (3) (4) (5)

Establish network connection with the mobile phone. Establish TCP connection with the mobile phone. Register the user. Send GPS location data to the mobile phone. The default state, according to the set cycle (default state is one frame every 5 s), the location data is uploaded to the mobile phone without interruption. (6) Network status detection and network quality testing, and according to the current situation of the network, to deal with the corresponding (such as broken line redial). (7) Receive network or third-party control or operation command and make corresponding processing (such as: GPS sampling cycle setting, stop sending GPS data, start sending GPS data, setting monitoring number, changing operation password, etc.). (8) Dealing with sudden alarm (automatic call center number, reporting police information and so on). The so-called “agreed SMS” can include: verify the legitimacy password, sender intention, information reply address, etc. For example, the first six digits of the message “334586EM” indicate the identity validity of the car owner, and the password is “334586”. The first English symbol “E” appeared later, indicating that the vehicle owner should inquire about the current parking position of the vehicle. The second letter “M” means “return the information to the owner’s mobile phone.”

6.5.2 Location Response Technology of Underground Parking Lot Vehicle location automatic response technology in the underground parking area is based on mobile operation network base station location technology. The technique is characterized by: (1) Use the vehicle-borne visual sensor system (see Fig. 2.1) to collect the image information around the parking position of the vehicle. (2) After the on-board signal processor processes the image information, the image information characteristic data around the vehicle parking position is output to the vehicle GPRS terminal through the RS232 interface. (3) The car owner uses the GPRS mobile phone to call the vehicle GPRS terminal number to realize the system connection operation. The vehicle GPRS terminal sends the image information characteristic data of the parking position around the signal processor to the owner GPRS mobile phone in short message mode.

256

6 Vehicle Auxiliary Intelligent Technology

Therefore, the owner can learn the location information of the vehicle from the SMS of the GPRS mobile phone.

6.5.2.1

System Composition

Vehicle position automatic response system in underground parking area firstly collects image information from the vehicle-mounted image sensor. The collected image signals are input to the signal processor through multiple channels. After processing and analyzing the image signal, the signal processor extracts the information features and outputs the characteristic data representing the vehicle position information to the vehicle GPRS terminal through the RS232 interface. Vehicular GPRS terminals communicate with GPRS mobile phone through GPRS backbone network (see Fig. 6.35).

6.5.2.2

Core Algorithm Steps and Principles

As mentioned above, in advance the onboard GPRS terminal must have access to the IP address and its corresponding terminal number authorization through the mobile communication operator. When the owner returns to the underground parking garage, but forgets his parking space and needs to find the location of the vehicle, it can also use the GPRS mobile phone to send the agreed SMS to the vehicle GPRS terminal, through the system network dialing, PPP negotiation, TCP connection and other operations initialization, and then look for and wait for the car GPRS terminal SMS reply. The difference between the ground parking position response technology and the vehicle position response technology in the underground parking lot lies in the fact that: In the former, the vehicle GPRS receives the GPS geographic location information data, while in the latter, the vehicle owner receives the image feature information data processed by the onboard signal processor, for example, parking spaces mark, etc. When the vehicle GPRS terminal receives the agreed SMS, the vehicle’s “automatic response system of the vehicle position in the underground parking lot” will faithfully execute the task, that is, the working procedure of vehicle location infor-

Vehicle mounted image sensor

GPRS

Signal processor

GPRS Fig. 6.35 Schematic diagram of GPRS mobile communication system

GPRS

6.5 Automatic Response Technique for Parking Location

257

Fig. 6.36 Ground parking sign map

Fig. 6.37 Edge detection of landmark on ground parking

( x1 , y1 )

( x2 , y2 )

( x3 , y3 )

( x4 , y4 )

mation collection, collation, analysis and feature extraction has been entered. For example, the agreed SMS “334586IM” means that the legal identity owner (password: 334586) requests the vehicle’s current environment view information feature (character: I) to reply to the owner’s mobile phone (character: M). Among them, the specific steps of view information feature extraction are as follows: (1) The image sensor (see Fig. 2.1) of the vehicle-mounted visual sensing system will collect the view around the vehicle in the order of front, back, left, right, top and bottom, such as the sign of the ground parking space shown in Fig. 6.36. (2) Automatic detection and determination of four corner points (xi , yi ), i  1, 2, 3, 4, as shown in Fig. 6.37. (3) Take the four corners (xi , yi ) (i  1, 2, 3, 4) of the parking space image area shown in Fig. 6.37 as four vertices. (4) In the target coordinate after the perspective transformation of  the proposed  image, four points are also taken, whose coordinates are xi , yi , i  1, 2, 3, 4, using the homogeneous coordinate transformation equation as follows

258

6 Vehicle Auxiliary Intelligent Technology

Fig. 6.38 Perspective transformation of views

Fig. 6.39 Regional screenshot and reinforcement

⎡ ⎤ ⎡ ⎤⎡ x ⎤ xi i a11 a12 a13 ⎢  ⎥ ⎣ a a a ⎦⎢ ⎥ y y 21 22 23 ⎣ i ⎦ ⎣ i⎦  a31 a32 a33 1 1

(6.66) ⎡

⎤ a11 a12 a13 Get the values of each parameter in the perspective matrix ⎣ a21 a22 a23 ⎦ a31 a32 a33 ' &   (5) The xi j , yi j of each pixel in the image is transformed into xi j , yi j in the target coordinate system after the perspective transformation by using the perspective matrix, that is ⎡

⎤⎡ x ⎤ ij a11 a12 a13 ⎢  ⎥ ⎣ a a a ⎦⎢ ⎥ 21 22 23 ⎣ yi j ⎦ ⎣ yi j ⎦  a31 a32 a33 1 1 xi j

⎤

⎡

(6.67)

Among them, i  1, 2, . . . , m, j  1, 2, . . . , n, m is the pixel column value, n is the pixel row value, thus obtains the beneficial recognition digital image, as shown in Fig. 6.38, therefore overcomes the distortion of image characteristic information caused by the squint of the image sensor. (6) “1, 2, 3,…, 9, 0” and 26 English capitalized characters are used as templates to find the symbol characters on the parking spaces, determine the minimum area of the characters and intercept them, as shown in Fig. 6.39.

6.5 Automatic Response Technique for Parking Location

259

(7) The minimum area graph of the intercepted character is processed with the following formula for grayscale processing Y  −0.299R + 0.587G + 0.144B

(6.68)

Can get a grayscale image (see Fig. 6.40). In formula (6.66): R, G and B represent the color three components of the RGB color space respectively, and Y is the gradation of the color space after the conversion of color space. (8) The method of maximum inter-class variance is used to two-valued images. The specific formula is 

dsti j  1 If sr ci j > thr eshold dsti j  0 If sr ci j ≤ thr eshold

(6.69)

The two-value processing of the image after grayscale processing is carried out, and the image is divided into two parts of the target and the background, as shown in Fig. 6.41. In formula (6.69), dsti j is the pixel of the two-valued image, sr ci j is the pixel of the gray image, and thr eshold is the set threshold. (9) The plane coordinate system ox y of the two-dimensional image is shown in Fig. 6.42, where x is the abscissa and y is the ordinate, and the origin of coordinates is o in the lower left corner of the image. (10) The image (see Fig. 6.42) is projected to the y axis (see Fig. 6.43). The length of the maximum projection width is defined as the length of the transverse cutting of the character, and the image is cut laterally at this length, as shown in Fig. 6.44 (11) The image after the transverse cutting (see Fig. 6.44) is projected to the x axis (see Fig. 6.45). Each character is cut longitudinally by using the gap between the numbers. The cutting results are shown in Fig. 6.46.

Fig. 6.40 Grayscale map of area screenshot Fig. 6.41 Two-valued diagram Fig. 6.42 Coordinate diagram

y 0

x

260

6 Vehicle Auxiliary Intelligent Technology

Fig. 6.43 y axis projection

Fig. 6.44 Transverse cut graph Fig. 6.45 x axis projection

Fig. 6.46 Longitudinal cut graph

Therefore, each character is cut off in the end. (12) To fill each of the cut characters so that all the character images are images of 11 × 17 pixel size. (13) Each character image after cutting is treated with a uniform scaling process to eliminate the horizontal deviation. That is, to avoid the horizontal deviation of the character deviating from the vertical direction of the camera or the side view, scaling every character image and normalizing it to the standard size of 16 × 32 size. And the gray level of the 0 area is defined as the background, the gray level of the 1 area is defined as a character. (14) According to the similarity between each character image and the template image, the character recognition of the car bit sign is realized. Among them, the similarity value adopts the template matching method, and the specific formula is I " J "

Sk 

( f (i, j)Pk (i, j))

i1 j1 I " J "

(6.70) Pk (i, j)

i1 j1

In this, Sk is the value of the similarity between the k‘th template and the current character image, I and J are the width and height of the number of character pixels, respectively. f (i, j) and Pk (i, j) are character pixels and template pixels, respectively. k is a template for the block k, k  1, 2, . . . , m, m is the total number of templates and takes m  36. When Sl > S0 and Sl  max{Sk }, the detection character is identified as a l’th l∈k

template. Among them, S0 is a similarity threshold, Sk is the value of the similarity between the k’th template and the current character image, 1 ≤ k ≤ m, m is the total number of modules, Sl is the maximum value of the similarity between the preset template and the current character image.

6.5 Automatic Response Technique for Parking Location

261

For example, the character “A285” shown in Fig. 6.46 is matched with the eleventh, second, eighth and fifth order templates, respectively, that is, the similarity values between the two corresponding characters and templates are all over the similarity threshold S0 , and each reaches the maximum value. (15) The signal processor, based on the view recognition result “A285”, outputs it to the vehicle GPRS terminal, the vehicle GPRS terminal immediately sends “A285” data to the owner’s GPRS cell phone by short message. The car owner can find the vehicle according to the parking information provided by the text message. Of course, it is necessary to point out that the above recognition results are obviously the adjacent parking signs on the front of the vehicle, which is automatically collected on the front of the vehicle. It is not the parking space that the vehicle itself stops, but it is the location sign closest to the vehicle being searched. Therefore, it enables the owner to find the nearest location of the vehicle easily.

Chapter 7

Functions of Laser Radar in Intelligent Cars

With the gradual maturity of automotive intelligent technology, the development of various sensing technologies and the significant decline of economic cost, the application of laser radar on smart cars has reached a fairly widespread level and is especially necessary for driverless vehicles.

7.1 Summary The Laser radar has been published in the 60s of the last century. A laser radar is a radar system that uses a laser beam to detect the location and speed of a target. The principle of laser radar is to launch a detection signal to the target (Laser beams), then the target reflected signal (target echo) is compared with the transmitted signal, and the relevant information of the target can be obtained after proper processing. For example, the parameters such as the target distance, orientation, height, speed, attitude, even shape, etc. Thus the object can be detected, tracked and identified. It is made up of a laser transmitter, a laser receiver and an information processing system. The laser is sent out by converting electrical pulses into light pulses. The laser receiver then converts the light pulse back from the target into an electric pulse. The laser can be divided into visible light wave, short wave infrared light, medium and long wave infrared light and long wave infrared light. The principle of laser radar is very close to that of the millimeter wave radar. However, the former uses laser as a signal source and laser pulses emitted by lasers to cause scattering to objects, such as trees, roads, bridges and buildings on the ground. A part of the light waves will be reflected on the receiver of the laser radar. According to the principle of laser range finding, the distance from the laser radar to the target point can be calculated. As for the radial velocity of the target can be determined by the Doppler frequency shift of the reflected light, besides it can also measure two or more distance, and calculate the rate of accelaration. This is the basic principle of the direct detection laser radar. © Shanghai Jiao Tong University Press, Shanghai and Springer Nature Singapore Pte Ltd. 2019 X. Zhang and M. M. Khan, Principles of Intelligent Automobiles, https://doi.org/10.1007/978-981-13-2484-0_7

263

264

7 Functions of Laser Radar in Intelligent Cars

When the target is continuously scanned by pulsed laser, the data of all the target points can also be obtained. After that, the accurate 3D image can be obtained by using this data. This is the basic principle of the laser imaging radar. Direct detection radar and laser imaging radar are all likely to be applied to intelligent cars or driverless cars. The former is used to detect the distance and speed of obstacles around the vehicle, and the latter is used for mapping (rendering) objects (scenery). Laser imaging radar is the product of the combination of laser radar and photoelectric imaging system. It is one of the most important achievements in the latest development of laser radar. Since the 80s of the last century, laser imaging radar has been developing rapidly. Now the 3D imaging laser radar can accurately give the target image with distance information (4D image), and even can give the target attitude and rolling speed etc. Laser imaging radar has been widely used in the fields of military, aviation, aerospace, and etc. The wavelength of the laser imaging radar is mainly 0.9, 1.06, 3.8 and 10.6 µm, which are several orders higher than that of microwave. Therefore, the speed resolution, measurement accuracy and antijamming performance of the laser imaging radar are all unparalleled by microwave radar. In the range of the laser imaging radar, the vast majority of the targets are distinguishable, which is the most significant feature of the laser imaging radar.

7.2 Working Principle of Laser Imaging Radar The key technologies of laser imaging radar include: high quality and controllable laser emission source, laser echo signal reception technology, two-dimensional or three-dimensional scanning technology, image processing and target recognition algorithm. The research of laser imaging radar is relatively early, and has been applied in the automatic navigation system of robot and target recognition. In recent years, some enterprises in China have begun to do this research. For example, the domestic SLAMTEC technology company has been engaged in research and development of robot related fields since 2009, and has nearly 7 years of research and development experience of robot autonomous localization and navigation algorithm, laser sensor and robot hardware system. At present, the RPLIDAR A1 and the upgraded version of the RPLIDAR A2 360° laser scanning range radar have been developed.

7.2.1 System Structure of Laser Imaging Radar See Fig. 7.1, which is a typical principle block diagram of the laser imaging radar. The whole system consists of 7 parts: a reference oscillator, a laser, an intensity modulator, a transmitter, a receiving device, a CCD array and a video processing unit. According to the part, it can be divided into 3 systems: Launch System (including reference

7.2 Working Principle of Laser Imaging Radar

265 Laser beam

Laser

Intensity modulator 1

Launch Reference oscillator

Intensity modulator 2

Laser signal gain amplifier Receive

Video processor

CCD array

Fig. 7.1 Schematic diagram of laser imaging radar

oscillator, laser, intensity modulator, launcher, receiving device), receiving system (CCD array) and information processing system (video processing subsystem). After the laser beam is modulated by intensity, it is launched from the launch system to the target that needs to be detected. The laser reflected from the target contains the physical information of the target (speed, position, surface condition, etc.). After the reflected laser signal is processed, the laser receiver can obtain the relevant information. This is the basic principle of laser radar.

7.2.2 Mathematical Model of Laser Radar It is assumed that P1 is the laser power emitted by a laser radar launcher, the laser beam has a uniform distribution of light intensity. The space stereoscopic angle is 1 , the angle between the direction of the emission and the normal section of the target section is θ (assuming that the radar is the target on the front, so θ is a sharp angle). The area of the laser beam and the target is A1 , the power of the laser obtained by the target is P

A1 cos θ P1 τa (R) R 2 Ω1

(7.1)

In the formula, τa (R) is the transmittance of the atmosphere when the laser transmission distance is R. Assuming that the target emission ratio is ρ, the solid angle of the scattering beam of the target is 2 , the effective area of the receiver is A2 (A2 < R 2 Ω2 ), and the receiver’s optical efficiency is τb , then the receiver’s laser power is P2  ρ P

A2 A1 cos θ A2 τb τa (R)  ρ 2 P1 τa (R) × 2 τb τa (R) R 2 Ω2 R Ω1 R Ω2

(7.2)

266

7 Functions of Laser Radar in Intelligent Cars

In general, the reflection of a target is diffuse reflection, and the reflected laser energy distributes in the whole space or half space. For the hemispherical space, Ω2  2π (sr), the formula (7.2) can also be expressed as P2  ρ

A1 cos θ A2 P1 τ22 (R) × τb 2 R Ω1 2π R 2

(7.3)

. When a laser radar system is used for large area detection, A2  2π R 2 , the formula (7.3) is simplified to P2  ρ

A1 cos θ P1 τ22 (R)τb R 2 Ω1

As a result, the detection distance of the laser can be expressed as  ρτb P1 A1 cos θ R  τ2 (R) P2 Ω1

(7.4)

(7.5)

At the same time, the maximum distance that the lidar can detect can be expressed as  Rmax  τ2 (R)

ρτb P1 A1 cos θ P2min Ω1

(7.6)

In the formula, P2min is the minimum laser power that can be received.

7.2.3 Target Characteristics and Laser Atmospheric Attenuation In order to make the imaging quality of the laser imaging radar meet certain requirements, it needs the target to have a good reflection property to the laser, which is the target characteristic. Objective and background are relative concepts, and the study of target characteristics is bound to study the background characteristics. The main physical quantities describing the characteristics of the target are the reflection ratio and the spectral reflectance. The former represents the reflection ability of the target to the full spectrum segment, and the latter characterizing the reflection ability of the target to a certain wavelength laser. For example, a laser with a wavelength of 900 nm, the spectral reflectance of a laser on a different object is shown as shown in Table 7.1. The transmission of laser imaging radar signals often need to go through the atmosphere, the atmosphere interacts with the laser signal, resulting in signal distortion, in addition, the effects of atmospheric dust, smoke, fog, rain and turbulence on the

7.2 Working Principle of Laser Imaging Radar

267

Table 7.1 Spectral reflectance ratio of some objects to 900 nm laser Target object Spectral reflectance ratio Dry, clean pine (accumulation)

0.94

Snow White masonry building

0.8–0.9 0.85

Limestone and clay

0.75

Smooth concrete Black rubber (synthetic rubber)

0.24 0.05

laser signal is greater, therefore there is a need to study the atmosphere on the laser signal attenuation. The discussion of atmospheric attenuation is very complicated. Under different circumstances, different components of the atmosphere have different results. There are usually no strict theoretical formulas. Most of the practical applications are based on experimental test formulas.

7.3 Imaging Scanning of Laser Imaging Radar The laser imaging radar is divided into two main types: the scanning laser imaging radar and the non scanning laser imaging radar.

7.3.1 Scanning Laser Imaging Radar Scanning laser imaging radar uses laser beam to scan targets quickly, and receives echo information of each part through the detector unit, so as to get the threedimensional image of the target. Figure 7.2 shows a schematic block diagram of a scanning laser imaging radar system. In the picture, the scan-detection unit is the core part of the system. The system sends a scanning signal in two directions of the X and Y axes of the right angle coordinate system. By detecting and receiving the echo signal after scanning, and then processing the signal through the processor, a two-dimensional image is formed at the end. With the movement of the laser imaging radar or the movement of the target, the scanning and detecting unit will continue to receive the two-dimensional image at the next time or another angle. Later a three-dimensional target and its image are constructed in the world coordinate system through the synthesis operation. Combining the imaging system with the distance measurement and velocity measurement system, the more accurate target characteristics can be obtained. According to Rayleigh criterion, for an optical system working at a wavelength of λ and an aperture of D, the limit of its resolution is

268

7 Functions of Laser Radar in Intelligent Cars Laser

Frequency stabilizing device

Modulator Optical antenna

x Scanner

Scanning control

Synchronous signal

y Detector

Preamplifier

Display and sound Data processor

Signal processor

3D imaging processor

To microcomputers

Fig. 7.2 Scanning laser imaging radar system block diagram

θ 

1.22λ D

(7.7)

If the laser beam is scanned in one direction, the maximum scanning angle is θ , the maximum number of single scan signals that can be sent out in a scan is N

Dθ θ  θ 1.22λ

(7.8)

If the radar is to be able to identify the target, N ≥ 12 must be satisfied. For the pulse laser radar, the maximum measurement distance Rmax limits the maximum value f max of the scanning pulse repetition frequency, i.e. f max ≤

c 2Rmax

(7.9)

An image is composed of row pixels and column pixels. Each pixel needs time to acquire. Therefore, the imaging time of the scanning system will be limited by many factors, which is the biggest drawback of scanning laser imaging radar. As shown in Fig. 7.3, it is a schematic of the internal structure of a scanning laser imaging radar.

7.3.2 Non-scanning Laser Imaging Radar The non scanning laser imaging radar has appeared in the 90s of last century. Non scanning laser imaging radar usually uses focal plane array device as photoelectric detector. Compared with scanning laser imaging radar, it does not need complex

7.3 Imaging Scanning of Laser Imaging Radar

269

Fig. 7.3 Schematic diagram of scanning laser imaging radar structure

Sine wave driver

Oscillator

Laser Diode

Square wave driver Filter mirror Image intensifier

Computer

CCD array Screen

Main Control Processor Photocathode

Fig. 7.4 Continuous phase non scanning laser imaging radar system

scanning device. Its speed, field of view and reliability of image acquisition have been greatly improved. At present, there are a variety of different technical schemes for non scanning laser imaging radar. For example, continuous wave phase method, FM continuous wave 4D imaging system, and focal plane array pulsed laser imaging radar, etc. The FM CW 4D imaging system has different ways to realize it. The following is a simple introduction to the continuous wave phase method (see Fig. 7.4). Among them, the signals generated by the oscillator are formed into sine and square waves respectively. The sine wave modulates the intensity of the semiconductor laser, and the square wave modulates the image intensifier. The emission part mainly includes the laser light source and the sinusoidal modulation circuit. The

270

7 Functions of Laser Radar in Intelligent Cars

receiving part mainly includes the microchannel image intensifier, the square wave modulation circuit and the CCD camera. The square wave modulation on the image intensifier can be achieved by applying a suitable square wave voltage on the photocathode. In this system, the laser emitted from the light source is irradiated on the target, and the laser is reflected through the target and is imaging in the photocathode by the lens. After that, the reflected wave is mixed with the voltage signal modulated by the same frequency in the image intensifier. The result of mixing is the output of a DC intensity field on the phosphor screen, which corresponds to the phase difference between the received optical signal and the modulated electrical signal, and then constitutes the photoelectric correlation array signal sequence. The signal sequence is then received by the CCD array. The obtained signal is converted to the A/D and then input to the microcomputer for image data processing. The average value of the echo signal received on the CCD array is the intensity information that can be obtained. The distance information of the target can be obtained by detecting the phase information in the echoed signal. The operating distance of the typical system can reach 1.5 km, the distance resolution is better than that of 15.2 cm, and the precision can reach 3 cm.

7.4 Common Vehicle Laser Radar The laser radar deployed on a smart car or an unmanned vehicle can be broadly divided into two major categories: laser detection radar and/or laser imaging radar.

7.4.1 Example of Laser Radar In general, smart cars are mainly equipped with visible and infrared cameras, and the laser detection radar is properly reconfigured (see Fig. 7.5). On the driverless vehicle, in order to improve the reliability and safety of the driving process, besides configuring visible light (and infrared) camera and millimeter wave radar, the laser imaging radar will also be configured.

Fig. 7.5 Example of laser detection radar

7.4 Common Vehicle Laser Radar

271

Fig. 7.6 Example of laser imaging radar

Figure 7.6 shows two kinds of laser imaging radar. Among them, Fig. 7.6a is a more common form of laser imaging radar. Figure 7.6b is a military night vision laser imaging radar.

7.4.2 Common Configuration of Unmanned Vehicles Although the technology of the driverless vehicle is not yet fully developed, many enterprises at home and abroad have begun to develop driverless cars, and are committed to improving their performance. Figure 7.7 shows the basic configuration of an unmanned vehicle developed by an enterprise in China. The video camera is used to detect the traffic lights, pedestrians, cyclists, stationary or moving obstacle and the vehicle may encounter in the process of moving. The laser range finder (that is, laser imaging radar) is used to accurately map the 3D images in the peripheral 200 m range. The millimeter wave radar, using the “three in the front and one in the back” configuration method, is used to detect fixed obstacles in a long distance. The micro sensor is also used to monitor whether the vehicle is deviated from the route designated by satellite navigation. This is realized by using the fusion algorithm of the vehicle direction and the vehicle position of the electronic compass and the gyroscope in the micro sensor. The former combines the north direction angle and the change rate measured by the electronic compass and the gyroscope, and estimates the true value of the north angle. The latter first synchronize the vehicle speed vector and the vehicle position vector measured by the Beidou satellite, and then estimate the vehicle location information according to the fusion results. The sensor signals are all delivered to the on-board microprocessor for data processing, calculation, storage, decision and instruction output. The display, sound and servo control mechanism can realize display output, sound prompt vehicle speed and state control under control instruction.

272

7 Functions of Laser Radar in Intelligent Cars

Fig. 7.7 Basic configuration diagram of unmanned vehicle

7.5 Advantages and Disadvantages of Laser Radar Although the application of Lidar in target detection has been quite mature and extensive, it has both advantages and disadvantages.

7.5.1 Advantages of Laser Radar First, the millimeter wave length is 1–10 mm, it is located in the wavelength range that overlaps between the microwave and the far infrared waves. Therefore, it has the characteristics of two kinds of spectrum. The theory and technology of millimeter wave are the extension of the microwave to high frequency and the development of the light wave to the low frequency. Microwave is an electromagnetic wave with a frequency of 300–300 GHz. It is an abbreviation for a limited frequency bandwidth in a radio wave. It is an electromagnetic wave between 1 mm and 1 m, which is a general name for decimeter wave, centimeter wave and millimeter wave. The frequency of the microwave is higher than that of the ordinary radio wave, which is usually also called “ultra high frequency electromagnetic wave”. Microwave is an electromagnetic wave with the wave particle like properties. The basic properties of microwave are usually presented as three characteristics of penetration, reflection and absorption. Compared with the ordinary millimeter wave radar and microwave radar, laser radar has a much higher frequency than the microwave frequency because it uses a laser beam. So it brings many advantages, which are as follows:

7.5 Advantages and Disadvantages of Laser Radar

273

(1) High resolution High angle, distance and velocity resolution can be obtained by laser radar. Generally, the angular resolution is not less than 0.1 mard, that is to say, it can distinguish two objects at 3 km distance with 0.3 m clearance from each other (which is impossible to achieve by microwave radar), and can track multiple targets at the same time. The distance resolution is up to 0.l m, and the speed resolution is within 10 m/s. The high resolution of distance and speed means that the distance based Doppler imaging technique can be used to obtain a clear image of the target. (2) Good concealment and strong ability to resist active interference The laser’s straight line propagation, good directivity and narrow beam can only be received on its propagation path. Therefore, the interception of the enemy is very difficult, and the laser radar’s launch system (the launch telescope) has a small caliber. The receiving area is narrow, and the probability of the intentional firing of the laser jamming signal into the receiver is very low. Besides, unlike microwave radar, which is susceptible to the widespread electromagnetic waves in nature, there are not many active signals that interfere with lidar in nature. Therefore, the ability of laser radar to resist active jamming is very strong and is suitable for working in the environment of increasingly complex and intense information warfare. (3) Good low altitude detection performance Because of the influence of the echo of various ground objects in the microwave radar, there is a blind area in a certain region (the undetectable area) in the low altitude. For the laser radar, only the irradiated target will produce reflection, and there is no effect of the ground echo.

7.5.2 Disadvantages of Laser Radar First, the laser radar is affected by the weather and the atmosphere at work. The laser is generally less attenuated and more distant in clear weather. In bad weather such as heavy rain, heavy smoke, thick fog and so on, the attenuation of laser power is greatly increased, and its propagation distance is greatly affected. For example, the CO2 laser with a working wavelength of 10.6 µm is a better choice for atmospheric transmission in all lasers, but the attenuation in bad weather is 6 times that of a clear day. The operating distance of CO2 lidar on the ground or low altitude is 10–20 km on sunny days, while in the bad weather, the transmission distance is reduced to less than 1 km. Moreover, the atmospheric circulation can also distort and shake the laser beam, which directly affects the measurement accuracy of the laser radar. Secondly, because the beam of the laser radar is very narrow, it is very difficult to search the target in space, and it will directly affect the interception probability and detection efficiency of the small target. The laser can only search and capture the target in a smaller range. Therefore, the laser radar is less directly applied to the battlefield for target detection and search.

274

7 Functions of Laser Radar in Intelligent Cars

Fig. 7.8 Full functional configuration of driverless car sensing

7.6 Perfect Configuration of Driverless Vehicle Sensors In view of the advantages and disadvantages of the laser radar, the function of the laser imaging radar does not fully cover the imaging performance advantages of the optical camera. As a result, an unmanned vehicle needs to configure an optical camera while configuring a laser radar (or a laser imaging radar). Conversely, in order to improve the stability, reliability and absolute safety of intelligent cars, we need to configure laser sensors, such as laser imaging radar, millimeter wave radar and other sensors, while configuring optical (infrared) cameras for driverless cars. In this way, the full - function technical goal of unmanned driving can be achieved. The full function sensor configuration of the driverless car is shown in Fig. 7.8. Once the vehicle has such a full functional sensing performance, reconfiguring the new neural network microprocessor and its corresponding software will be the process of vehicle and other transportation tools being applied to reality. The automotive intelligent technology and even the unmanned vehicle are the new interdisciplinary fields. It’s like a virgin land waiting to be reclaimed. Although it is a rich land and full of vitality, it will take a long time to make this virgin land become a vigorous granary. There is no doubt that there are many new theories and techniques in this new and typical interdisciplinary technology field, which need people to explore. To be sure, in the near future, people will be able to see new, reliable, and comfortable intelligent driverless cars going deep into people’s lives. This is the end of the book, too.

References

1. 2. 3. 4. 5. 6. 7. 8. 9. 10.

11.

12. 13. 14. 15. 16. 17.

World Health Statistics (2008) Wold heath organizations Zhang X, Ying J (2010) Artificial intelligence vision. China Water Conservancy and Hydropower Press, Beijing Zhang X, Ying J (2009) Intelligent Detection of Artificial Visual Perception. Science Press, Beijing Zhang X, Jiao D, Wang H et al (2009) Intelligent device for vehicle automatic identifying road deep pit and obstacle. China Invention Patent CN200910049885.3, Apr 2009 Zhang X, Jiao D, Ying J et al (2009) Intelligent method for vehicle automatic identifying road deep pit and obstacle. China Invention Patent CN200910049884.9, Apr 2009 Zhang X, Ying J, Hu Z et al (2010) Intelligent control device and method for safety vehicle distance. China Invention Patent CN201010283951.6, Sep 2010 Zhang X, Ying J, Cui C et al (2010) An intelligent device and method for preventing the collision of pedestrians from cars. China Invention Patent CN201010023042.9, Jan 2010 Zhang X, Ying J, Mo Z et al (2008) Vehicle intelligent rear view device. China Invention Patent CN200810033985.2, Feb 2008 Zhang X, Ying J, Mo Z et al (2008) Vehicle intelligent rear view method. China Invention Patent CN200810033985.2, Feb 2008 Zhang X, Xu L, Cheng Y et al (2009) An intelligent device for vehicle automatic identification of road speed limit signs. China Invention Patent CN200910047471.7, Mar 2009 Zhang X, Jiao D, Ying J et al (2009) An intelligent method for vehicle automatic identification of road speed limit signs. China Invention Patent CN200910047472.1, Mar 2009 Zhang X, Huang D, Zou X et al (2013) An intelligent technology to prevent and control the vehicle rear-end. Microcomput Appl 29(11):20–22 (In Chinese) Ji C, Zhang X, Ying J et al (2006) PRIC Key Technologies used in storage yard based on machine vision. WSEAS Trans Inf Sci Appl 3(7):1341–1346 Zhang W, Gao X, Zhang X (2013) Different modality image registration algorithm according to the block information. Microcomput Appl 29(1):58–60 (In Chinese) Ying J, Zhang X, Ji C et al (2006) Multidimensional decision-making using composed fuzzy method in images matching. Syst Eng Electron 28(12):1797–1819 (In Chinese) Zhang X, Khan MM (2012) Intelligent visual perception tutorial. China Water Conservancy and Hydropower Press, Beijing Zhang X, Ying J, Shi Z et al (2010) Intelligent control system of traffic signal and its control method. China Invention Patent CN201010023041.4, Jan 2010

© Shanghai Jiao Tong University Press, Shanghai and Springer Nature Singapore Pte Ltd. 2019 X. Zhang and M. M. Khan, Principles of Intelligent Automobiles, https://doi.org/10.1007/978-981-13-2484-0

275

276

References

18.

Zhang X, Ying J, Wang S et al (2010) Intelligent monitoring device and method for drunk driving. China Invention Patent CN201010299236.1, Oct 2010 Zhang X, Zhang X, Chen T et al (2007) A device to prevent error from stepping on the brakes. China Invention Patent CN200710042893.8, June 2007 Zhang X, Zhang X, Zhu X et al (2007) Automatic control method for preventing wrong stepping on brakes. China Invention Patent CN200710042894.0, June 2007 Zhang X, Ma L, Ying J et al (2010) Intelligent device and method for monitoring the driving of two hands out of steering wheel. China Invention Patent CN201010501538.2, Oct 2010 Ying J, Zhang X, Ma L et al (2010) Intelligent identification device and method for driver’s fatigue state. China Invention Patent CN201010284829.0, Sep 2010 Li X, Zhang X, Wu H et al (2010) Intelligent identification device and method for abnormal state of locomotive engine and transmission mechanism. China Invention Patent CN201010522633.0, Oct 2010 Zhang X, Ying J, Zhang X et al (2010) Intelligent switch device and method of car door. China Invention Patent CN201010278102.1, Sep 2010 Zhang X, Ying J, Wang S et al (2010) Automatic response system and method of vehicle location in underground parking lot. China Invention Patent CN 201010501388.5, Oct 2010 Zhang X, Zhu L, Su Y et al (2007) Automatic Monitoring system of Bridge load limit. China Invention Patent CN200710047398.4, 25 Oct 2007 Zhang X, Ying J, Zhou C et al (2007) Automatic monitoring method of bBridge load limit. China Invention Patent CN200710047395.0, Oct 2007 Ji C (2007) Study of image information technology based on PRIC theory. Dissertation, Shanghai Jiao Tong University Hu Z (2004) research and implementation of bomb guidance technology. Dissertation, Shanghai Jiao Tong University Sun Z, Zhang X, Zhang F et al (2003) Fast recognition and tracking of moving targets in complex background. Microelectron Comput 11:48–51 (In Chinese) Xiao H, Zhang X, Zeng G et al (2005) Auto recognition method for automobile overloading. China Invention Patent CN200510025067.1, Apr 2005 Ji C, Zhang X, Zeng G et al (2005) Auto recognition system for automobile overloading. China Invention Patent CN200510024928.4, Apr 2005 Ma L (2006) Research on measurement of moving images based on video and method of sense. Hefei University of Technology. Dissertation, Hefei Industrial University Zhang X, Ying J, Zhang X et al (2010) Intelligent device and method of bus special seat. China Invention Patent CN201010501545.2, Oct 2010 Forsyth DA, Ponce J (2004) Computer vision: a modern approach. Prentice Hall, Upper Saddle River Shapiro LG, Stockman GC (2005) Computer vision. Addison Wesley, New Jersey Snyder WE, Qi H (2005) Machine vision. Cambridge University Press, Cambridge Li J (1991) Theory and practice of computer vision. Shanghai Jiao Tong University Press, Shanghai Sun J (2001) Detection and control technology based on computer vision. Dissertation, Shanghai Jiao Tong University Ji C, Zhang X, Ying J et al (2006) Real time industry im age detecting based on PC 104. Microcomput Inf 22(10):212–214 (In Chinese) Li Y, Zhang X (2004) Application of video processing chip SAA7111AWP in high speed video sampling system. Electron Tech 5:24–26 (In Chinese) Sun J (2007) The study of dynamic capturing technology in video monitor. Dissertation, Shanghai Jiao Tong University Sun J, Zhang X, Jiang L (2000) Two methods of real-time image capture under Windows. Microcomput Appl 16(5):40–43 (In Chinese)

19. 20. 21.

22. 23.

24. 25.

26. 27. 28. 29. 30. 31. 32. 33. 34. 35. 36. 37. 38. 39. 40. 41. 42. 43.

References 44. 45. 46. 47. 48. 49. 50. 51. 52. 53. 54. 55. 56. 57. 58. 59. 60. 61. 62. 63. 64. 65. 66. 67. 68. 69.

70.

277

Sun J, Zhang X, Jiang L (2000) Image acquisition and display technology based on Windows 98. Microcomput Appl 16(6):12–14 (In Chinese) Zhang Y (2009) Automatic image stitching method based on feature points. Dissertation, Shanghai Jiao Tong University Ying J, Zhang X, Mo Z et al (2008) A method of automatic mosaic of wide angle images. China Invention Patent CN200810036149.X, Apr 2008 Rosenfeld A, Johnston E (1973) Angle detection on digital curves. IEEE Trans Comput 24 (9):940–941 Zeng G, Zhang X, Hu Z et al (2003) Image threshold segmentation method based on fuzzy degree. Microcomput Appl 19(2):10–11 (In Chinese) Jahne B (1997) Digital image processing. Springer, New York Wu J (2005) Image recognition algorithm and real-time processing technology. Dissertation, Shanghai Jiao Tong University Meyer F (1992) Color image segmentation. In: Proceedings of the international conference on image processing and its applications, pp 303–306 Serra J (1982) Image analysis and mathematical morphology. Academic Press Inc, Orlando Canny J (1986) A computational approach to edge detection. IEEE Trans Pattern Anal Mach Intell 8(6):679–698 Zhou C (2009) Research on crutial intelligent and fast identification technology of medical clinical image. Dissertation, Shanghai Jiao Tong University Jiang Y (2006) Fast target recognition algorithm and implementation based on image algebraic features. Dissertation, Shanghai Jiao Tong University Hu Z (2006) Optimization algorithm and implementation of target recognition based on gray gradient feature. Dissertation, Shanghai Jiao Tong University Hu MK (1962) Visual pattern recognition by moment invariants. IRE Trans Inf Theory 8 (2):179–187 Iivarinen J, Peura M, Srel J, Visa A (1997) Comparison of combined shape descriptors for irregular objects. In: 8th British Machine Vision Conference, BMVC’97 Alkhansari M (2001) A fast globally optimal algorithm for template matching using low-resolution pruning. IEEE Trans Image Process 10(4):526–533 Men P (2005) Target recognition and tracking based on composite algorithm. Dissertation, Shanghai Jiao Tong University Sun Z (2005) Research and implementation of target recognition and tracking algorithm. Dissertation, Shanghai Jiao Tong University Ji C, Zhang X, Wu H et al (2007) Fast image matching based on gray value weighting NMI. Comput Eng 33(24):25–28 (In Chinese) Jiang Y, Zhang X, Hu Z (2005) Multiple algorithm of target recognition based on singular value features. Comput Eng 31(15):151–153 (In Chinese) Men P, Zhang X, Zhang F et al (2004) Tagret recognition and tracking based on NMI feature. Microcomput Inf 20(3):24–26 (In Chinese) Wu J, Zhang X, Zhang F et al (2004) The fats reorgranization arithmetic of motive target. Microcomput Inf 20(3):27–28 (In Chinese) Men P, Zhang X, Zhang F et al (2004) Study on methods about disposal and ricognition of image. Microcomput Inf 20(5):103–105 (In Chinese) Wu J, Zhang X, Zhang F et al (2004) The experiment research of adge detection in digital image. Microcomput Inf 20(5):106–107 (In Chinese) Ying J, Zhang X, Men P et al (2004) A method of automatic recognition and fast tracking of target images. China Invention Patent CN200410068030.2, Nov 2004 Ying J, Zhang X, SUN Z et al (2004) Methods to reduce the target image automatic identification and rapid tracking error rate. China Invention Patent CN200410084298.5, Nov 2004 Ying J, Zhang X, Wu J et al (2004) A method to Improve the automatic recognition and tracking speed of target images. China Invention Patent CN200410089272.x, Dec 2004

278

References

71.

Ying J, Zhang X, Men P et al (2004) A piecewise matching method for target image recognition and tracking. China Invention Patent CN200410103042.4, Dec 2004 Chen Y (2009) Design of automatic adjustment method of tv camera color intelligent. Dissertation, Shanghai Jiao Tong University Zhang X, Ying J, Cheng Y et al (2008) Automatic Adjustment method of tv camera color intelligent. China Invention Patent CN200810201622.5, Oct 2008 Zhang X, Xu L, Dong C et al (2008) Automatic adjustment device of tv camera color intelligent. China Invention Patent CN200810201620.6, Oct 2008 Zhang X, Ying J, Guo J et al (2007) Intelligent control method of platform shield door. China Invention Patent CN200710045172.0, Aug 2007 Guo J, Zhang X, Ying J et al (2007) Intelligent control system of platform shield door. China Invention Patent CN200710045171.6, Aug 2007 Trucco E, Verri A (1998) Introductory techniques for 3-D computer vision. Prentice Hall, Upper Saddle River Zhang Z (1997) Parameter estimation techniques: A tutorial with application to conic fitting. Image Vis Comput J 15(1):59–76 Ji C, Zhang X, Xiao H et al (2005) Automatic measurement method of computer vision system in material field measurement. China Invention Patent CN200510026200.5, May 2005 Ji C, Zhang X, Xiao H et al (2005) Automatic measurement system of computer vision system in material field measurement. China Invention Patent CN200510026197.7, May 2005 Qin Z (2004) 3D target recognition based on model for single frame image. Dissertation, Shanghai Jiao Tong University Zhang X, Mo Z, Jiao D et al (2009) Monitoring system and method for anti rollover of steel roll step shift. China Invention Patent CN 200910056331.6, Aug 2009 Zhang X, Ying J, Jiao D et al (2009) Automatic detection method of particle size distribution. China Invention Patent CN200910056329.9, Aug 2009 Bradski G, Kaehler A (2008) Learning opencv: computer vision with the opencv library. O’Reilly Media Inc, Sebastopol Zhang X, YING Junhao, MA Li et al (2010) Automatic identification method of billet number. China Invention Patent CN201010023040.X, Jan 2010 Zhang X, Dai Y, Zhu L et al (2012) Control method of fixed length cutting for multi flow billet. China Invention Patent CN201210510425.8, Dec 2012 Shults RA, Sleet DA, Elder RU (2002) Association between state level drinking and driving countermeasures and self reported alcohol impaired driving. Int J Press Vessels Pip 8:106–110 James M (2003) Drinking and driving. CAMJ 168(3):313 Yang L (2005) Preparation of ferrate compound oxide gas sensitive material and study of new gas sensor. Dissertation, Kunming University of Science and Technology Zhang Y (2007) Intellectualized technique in moderm automobile industry. Hunan Agric Mach 9:153–154 (In Chinese) Zhu Z (2004) Research of intelligent control for integrated automobile powertrain system. Dissertation, Anhui Agriculture University Zheng Z (2009) Vehicle radio sensor network and intelligent traffic. Promot Inf Technol 12:21–22 (In Chinese) Yang D, Li K, Zheng S et al (2003) Vehicle technologies in intelligent transport systems. Automobile Eng 25(3):223–226 (In Chinese) Lu X, Lu H, Zhang F (2005) The design and application of GPRS and GPS in vehicular information service system. Microcomput Inf 21(3):188–189 (In Chinese) Ming P, Yang L, Zhang L et al (2005) Intelligent vehicle navigation and wireless interconnection system based on 3G technology. J Wuhan Univ Technol 27(11):66–68

72. 73. 74. 75. 76. 77. 78. 79.

80.

81. 82. 83. 84. 85. 86. 87.

88. 89. 90. 91. 92. 93. 94. 95.

References 96.

97. 98. 99. 100. 101. 102. 103.

279

Rovati FS, Gardella P, Zambotti P Pau D (2003) Spatial-temporal motion estimation for image reconstruction and mouse functionality with optical or capacitive sensors. In: IEEE transactions on consumer electronics, vol 49, Aug 2003, pp 711–718 Fons M, Fons F, Canto E (2006) Design of an embedded fingerprint matcher system. In: IEEE tenth international symposium on consumer electronics, St. Petersburg, pp 1–6 Maltoni D, Maio D, Jain AK et al (2003) Handbook of fingerprint recognition. Springer Verlag, New York, pp 1–9 Zhang X, Lu D, Huang D et al (2014) Real Time Information incremental method of navigation map. GEOMATICS WORLD 21(3):84–92 (In Chinese) Tang L (2004) Research on fingerprint recognition algorithm and its application. Dissertation, Shanghai Jiao Tong University Zhou L, Zhang Y, Wang P (2008) Design of embedded fingerprint identification system based on swipe sensor. Comput Eng Des 29(17):4401–4403 (In Chinese) Yu J (2015) Research on ground target recognition algorithm for LADAR range imagery. Dissertation, Beijing Institute of Technology Liu S, Hu H (2017) Development trend of foreign laser radar. Ship Electron Eng 37(6):1–4 (In Chinese)