Author Topic: Design Patterns for Deep Learning Architectures  (Read 251 times)

0 Members and 1 Guest are viewing this topic.

Offline Flavio58

Design Patterns for Deep Learning Architectures
« Reply #1 on: May 16, 2018, 09:56:42 PM »
Advertisement




Deep Learning Architecture can be described as a new method or style of building machine learning systems. Deep Learning is more than likely to lead to more advanced forms of artificial intelligence. The evidence for this is in the sheer number of breakthroughs that had occurred since the beginning of this decade. There is a new found optimism in the air and we are now again in a new AI spring. Unfortunately, the current state of deep learning appears too many ways to be akin to alchemy. Everybody seems to have their own black-magic methods of designing architectures. The field thus needs to move forward and strive towards chemistry, or perhaps even a periodic table for deep learning. Although deep learning is still in its early infancy of development, this book strives towards some kind of unification of the ideas in deep learning. It leverages a method of description called pattern languages.
[/size]Pattern Languages are languages derived from entities called patterns that when combined form solutions to complex problems. Each pattern describes a problem and offers alternative solutions. Pattern languages are a way of expressing complex solutions that were derived from experience. The benefit of an improved language of expression is that other practitioners are able to gain a much better understanding of the complex subject as well as a better way of expressing a solution to problems.[/color]
[/size]The majority of literature in the computer science field, the phrase “design patterns” is used rather than “pattern language”. We purposely use “pattern language” to reflect that the field of Deep Learning is a nascent, but rapidly evolving, field that is not as mature as other topics in computer science. There are patterns that we describe that are not actually patterns, but rather may be fundamental. We are never certain which will are truly fundamental and only further exploration and elucidation can bring about a common consensus in the field. Perhaps in the future, a true design patterns book will arise as a reflection of the maturity of this field.[/color]
[/size]Pattern Languages were originally promoted by Christopher Alexander to describe the architecture of businesses and towns. These ideas where later adopted by object oriented programming (OOP) practitioners to describe the design of OOP programs. The seminal book “Design Patterns” by the GoF is evidence of the effectiveness of this approach. Pattern languages were extended further into other domains such as user interfaces, interaction design, enterprise integration, SOA and scalability design.[/color]
[/size]In the domain of machine learning (ML) there is an emerging practice called “Deep Learning”. In ML there are many new terms that one encounters such as Artificial Neural Networks (ANN), Random Forests, Support Vector Machines (SVM) and Non-negative Matrix Factorization (NMF). These however usually refer to a specific kind of machine learning algorithm. Deep Learning (DL) in contrast is not really one kind of algorithm, rather it is a whole class of algorithms that tend to exhibit similar characteristics. DL systems are ANN that are constructed with multiple layers (sometimes called Multi-level Perceptrons). The idea is not entirely new, since it was first proposed back in the 1960s. However, interest in the domain has exploded with the help of advancing computational technology (i.e. GPU) and bigger training data sources. Since 2011, DL systems have been exhibiting impressive results in the field of machine learning.[/color]
[/size]The confusion with DL arises when one realizes that there actually many algorithms and it is not just a single kind. We find the conventional Feed forward Networks also known as Fully Connected Networks (FCN), Convolution Networks (ConvNet), Recurrent Neural Networks (RNN) and less used Restricted Boltzmann Machines (RBM). They all share a common trait in that these networks are constructed using a hierarchy of layers. One common pattern for example is the employment of differentiable layers, this constraint on the construction of DL systems leads to an incremental way of evolving the machine into something that learns classification. There are many patterns that have been discovered recently and it would be fruitful for practitioners to have at their disposal a compilation of these patterns.[/color]
[/size]Pattern languages are an ideal vehicle for describing and understanding Deep Learning. One would like to believe the Deep Learning has a solid fundamental foundation based on advanced mathematics. Most academic research papers will conjure up high-falutin math such as path integrals, tensors, hilbert spaces, measure theory etc. but don't let the math distract oneself from the reality that our collective understanding remains minimal. Mathematics you see has its inherent limitations. Physical scientists have known this for centuries. We formulate theories in such a way that the expressions are mathematically convenient. Mathematically convenience means that the math expressions we work with can be conveniently manipulated into other expressions. The Gaussian distribution for example is ubiquitous not because its some magical construct that reality has gifted to us. It is ubiquitous because it is mathematically convenient.[/color]
[/size]Pattern languages have been leveraged in many fuzzy domains. The original pattern language revolved around the discussion of architecture (i.e. buildings and towns). There are pattern languages that focus on user interfaces, on usability, on interaction design and on software process. These all don't have concise mathematical underpinnings yet we do extract real value from these pattern languages. In fact, the specification of a pattern language is not too far off from the creation of a new algebra or a category theory in mathematics. Algebras are strictly consistent but they are purely abstract and may not need to have any connection with reality. Pattern languages are however connected with reality, with their consistency rules are more relaxed than an algebra. In our attempt to understand the complex world of machine learning (or learning in general) we cannot always leap frog into mathematics. The reality may be such that our current mathematics are woefully incapable of describing what is happening.[/color]
[/size]An additional confusion that many machine learning practitioners will encounter when they are first presented with this idea of 'patterns' is that they mistakenly associate it with usual use of the word 'pattern' in their own field. Machine learning involves the development of algorithms that perform pattern recognition. So when you google deep learning with patterns, you will find literature that covers the subject of pattern recognition. This book is not about pattern recognition in the conventional machine learning sense.[/color]
[/size][/size][size=78%]https://docs.google.com/document/d/1ce_uZagVuqKRHkBqvdsESCJYdKmwS3tnLeTNFiWVjmc/edit?usp=sharing[/size][/size]

http://www.deeplearningpatterns.com/doku.php?id=overview


Consulente in Informatica dal 1984

Software automazione, progettazione elettronica, computer vision, intelligenza artificiale, IoT, sicurezza informatica, tecnologie di sicurezza militare, SIGINT. 

Facebook:https://www.facebook.com/flaviobernardotti58
Twitter : https://www.twitter.com/Flavio58

Cell:  +39 366 3416556

f.bernardotti@deeplearningitalia.eu

#deeplearning #computervision #embeddedboard #iot #ai

 

Related Topics

  Subject / Started by Replies Last post
0 Replies
263 Views
Last post May 24, 2018, 01:42:00 AM
by Flavio58
0 Replies
99 Views
Last post June 05, 2018, 07:13:20 PM
by Flavio58
0 Replies
167 Views
Last post June 10, 2018, 02:52:56 AM
by Flavio58
0 Replies
90 Views
Last post October 25, 2018, 08:09:44 PM
by Flavio58
0 Replies
53 Views
Last post October 30, 2018, 10:02:29 AM
by Flavio58

Sitemap 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326