9/7/2014
Convol uti onal Neur al Networ k Wor kbench - CodePr oj ect 10,720,671 members members (82,205 on line)
home
articles
qui ck answers
di scussi ons
M em emb er er 9 96 96 07 07 92 92
features
commu comm unity
help
Sign out
590
Search for articles, questions, tips
Articles » General Programming » Algorithms & Recipes » Neural Networks
Article
Browse Code Bugs / Suggest Suggestions ions
Convolutional Neural Network Network Workbench Workbench Filip D'haene , 8 Jul 2014
Rate:
4.93 (87 votes)
Stats Revisions (195)
About Article
A workbench to create, train, and test convolutional neural networks against against the MNIST and CIFAR-10 datasets
Alternatives Alter natives Comments (173)
Download CNNWB Sources Download Setup
Type
Article
Licence
CPOL
First Posted
28 Dec 2010
Views
200,281
Bookmarked
260 times
C# . NE NET
Introduction
View this article's View this artic le's Workspace Wor kspace
Fork this Workspace
Add your own Add your alternative alter native version
Dev WPF
This article article is about a framework in C# 4.0 that allows to create, train, and test convolutional neural networks networks against the MNIST and the CIFAR-10 dataset of 10 diff different erent natural objects. objects. I initially based me on an article by Mike O'Neill on the The Code Project an Project and d grad gradual ually ly adde added d new features that I've I'v e f ound ound interesting in research documents found on the internet. internet. Dr. Yann LeCun's LeCun's pa paper: per: Gr Gradie adient-Based nt-Based Learning Applied to Document Recognition is Recognition is a great paper to get a better understanding of the principles of convolutional neural networks and the reason why they are so successful in the area of machine vision.
Windows Vi su su al al -S -St ud udi o QA
XAML
Beginner
Intermediate , +
The Th e Code The main goal of this project was to build a more flexible flexible and ext exten end dabl able e managed version of Mike O'Neill's excellent C++ project. I've included and used the splendid WPF TaskDialog Wrapper from Sean A. Hanley, Hanley, the Extended WPF Toolkit and Toolkit and for unzipping the CIFAR-10 dataset the open-source SharpDevelop SharpZipLib module. module. Visual Studio 2012/2013 and 2012/2013 and Windows 7 are the minimum requirements. I made maximal use of the parallel functionality offered in C# 4.0 by letting the user at all times choose how many logical cores are used in the parallel optimized code parts with a simple manipulation of a sliderbar next to the View combobox.
Using the Code Here is the example code to construct a LeNet-5 network in code (see the InitializeDefaultNeuralNetwork() function in MainViewWindows.xaml.cs ): Collapse | Copy Code
NeuralNetwork cnn = new new NeuralNetwork NeuralNetwork (DataProvider, "LeNet-5" "LeNet-5", , 10 10, , 0.8D, LossFunctions.MeanSquareError, DataProviderSets.MNIST, TrainingStrategy.SGDLevenbergMarquardt, 0.02D); cnn.AddLayer(LayerTypes.Input, 1, 32 32, , 32 32); ); cnn.AddLayer(LayerTypes.Convolutional, ActivationFunctions.Tanh, 6, 28 28, , 28 28, , 5, 5); cnn.AddLayer(LayerTypes.AveragePooling, ActivationFunctions.Tanh, 6 , 14 14, , 14 14, , 2 , 2); bool[] bool [] maps = new bool bool[ [6 * 16 16] ] { true, true , false false, ,false false, ,false false, , true true, , true true, , true true, , false false, ,false false, ,true true, , true true, , true true, , true true, , false false, ,true true, , true true, , true, true , true true, , false false, ,false false, , false false, ,true true, , true true, , true true, , false false, ,false false, ,true true, , true true, , true true, , true true, , false false, ,true true, , true, true , true true, , true true, , false false, , false false, ,false false, ,true true, , true true, , true true, , false false, ,false false, , true true, , false false, ,true true, , true true, , true true, , false, false ,true true, , true true, , true true, , false false, ,false false, ,true true, , true true, , true true, , true true, , false false, , false false, ,true true, , false false, ,true true, , true true, , false, false ,false false, ,true true, , true true, , true true, , false false, ,false false, ,true true, , true true, , true true, , true true, , false false, ,true true, , true true, , false false, ,true true, , false, false ,false false, ,false false, ,true true, , true true, , true true, , false false, ,false false, ,true true, , true true, , true true, , true true, , false false, ,true true, , true true, , true }; cnn.AddLayer(LayerTypes.Convolutional, ActivationFunctions.Tanh, 16 16, , 10 10, , 10, 10 , 5 , 5, new Mappings(maps)); cnn.AddLayer(LayerTypes.AveragePooling, ActivationFunctions.Tanh, 16 16, , 5, 5, 2, 2); cnn.AddLayer(LayerTypes.Convolutional, ActivationFunctions.Tanh, 120 120, , 1, 1, 5, 5); cnn.AddLayer(LayerTypes.FullyConnected, ActivationFunctions.Tanh, 10 10); ); cnn.InitializeWeights();
Related Articles Neural Network for Recognition of Handwritten Digits A Neural Network on GPU Online handwriting recognition using multi convolution neural networks Large pattern recognition
http://www.codepr oj oj ect.com/Ar ti cl es/140631/Convol uti onal - Neur al- N etwor k- MN MNIST- Wor kbench
1/8
9/7/2014
Convolutional Neural Network Workbench - CodeProject
Design View
system using multi neural networks Multiple convolution neural networks approach for online handwriting recognition Neural Network for Recognition of Handwritten Digits in C# Neural Networks on C# An Introduction to Encog Neural Networks for C# AForge.NET open source framework Designing And Implementing A Neural Network Library For Handwriting Detection, Image Analysis etc.- The BrainNet Library - Full Code, Simplified Theory, Full Illustration, And Examples AI: Neural Network for Beginners (Part 3 of 3) Brainnet 1 - A Neural Netwok Project - With Illustration And Code - Learn Neural Network Programming Step By Step And Develop a Simple Handwriting Detection System C# Application to Create and Recognize Mouse Gestures (.NET) An Introduction to Encog Neural Networks for Java Library for online handwriting recognition system using UNIPEN database. AI : Neural Network for beginners (Part 1 of 3)
In Design View, you can see how your network is defined and get a good picture of the current distribution of weight values in all the layers concerned.
Training View
Mouse gestures recognition Creating Optical Character Recognition (OCR) applications using Neural Networks Neural Cryptography AI : Neural Network for beginners (Part 2 of 3)
Related Research
Gartner: Magic Quadrant for On-Premises Application Platforms
Expanding active decisionmaking: The power of integrating business rules and events
In Training View you can train the network. The 'Play' button gives you the 'Select Training Parameters' dialog where you define all the training parameters. The 'Training Scheme Editor' button gives you the possibility to make training schemes to experiment with. At any time the training can be paused or aborted. The 'Star' button will forget (reset) all
http://www.codepr oject.com/Articles/140631/Convolutional-Neural-Network- MNIST-Workbench
Toad Oracle: Tips to Simplify Database Admi nistration and Development
2/8
9/7/2014
Convolutional Neural Network Workbench - CodeProject the learned weight values.
Learn Agile: Ten Tips for Launching and Testing High Quality Apps for the American Market
Testing View
http://www.codepr oject.com/Articles/140631/Convolutional-Neural-Network- MNIST-Workbench
3/8
9/7/2014
Convolutional Neural Network Workbench - CodeProject
In Testing View you get a better picture of the testing (or training samples) which are not recognized correctly .
Calculate View
In Calculate View we test a single testing or training sample with the desired properties and get a graphical view of all the output values in every layer.
http://www.codepr oject.com/Articles/140631/Convolutional-Neural-Network- MNIST-Workbench
4/8
9/7/2014
Convolutional Neural Network Workbench - CodeProject
Final Words I would love to see a GPU integration for offloading the highly parallel task of learning the neural network I made an attempt to use a simple MVVM structure in this WPF application. In the Model folder you find the NeuralNetwork and DataProvider class which provide all the neural network code and deals with loading and providing the necessary MNIST and CIFAR-10 training and testing samples. Also a NeuralNetworkDataSet class is used to load and save neural network definitions. The View folder contains four different PageViews and a global PageView which acts as the container for all the different views ( Design, Training, Testing and Calculate). Hope there's someone out there who can actually use the code and improve on it. Extend it with an unsupervised learning stage for example (encoder/decoder construction), implement better loss-functions and more training strategies (conjugate gradient, l-bgfs, ...), add more datasets and use new activation fuctions, etc.
History 1.0.3.7: (07-08-14)
BugFix: slow speed resolved in Testing view Added SGDLevenbergMarquardtModA training strategy. This can be used with a softmax output layer Posibility to save the weights while training. Just click on Pause and then Save/Save as... Various smaller fixes and optimizations 1.0.3.6: (05-02-14)
Choice between four Training Strategies: SGDLevenbergMarquardt SGDLevenbergMarquardtMiniBatch SGD SGDMiniBatch BugFix: Derivative of ReLU's activation functions Added SoftSign activation function Overall 20% more training performance over previous version Faster binary save of the network weights Various smaller fixes and optimizations 1.0.3.5: (03-13-14)
StochasticPooling and L2Pooling layers now correctly implemented Native C++ implementation + managed C++/CLI wrapper 1.0.3.4: (12-10-13)
Support for Stochastic Pooling layers Much faster binary load and save of a cnn ReLU activation function now working as expected 1.0.3.3: (11-26-2013)
Bugfix: Download datasets now working as expected Bugfix: Softmax function corrected Bugfix: DropOut function corrected 1.0.3.2: (11-15-2013)
Bugfix: Local layer and Convolutional layer now works properly Bugfix: Cross Entropy loss now works better (in combination with a SoftMax activation function in a final fully connected layer) Added LeCun's standard LeNet-5 tr aining scheme 1.0.3.1: (11-09-2013)
Now the last min/max display preference is saved Added some extra predefined training parameters and schemes Bugfix: Average error not showing correctly after switching between neural networks with a different objective function Bugfix: Sample not always showing correctly in calculate view Bugfix: The end result in testing view is not displaying the correct values 1.0.3.0: (11-06-2013)
Supports dropout Supports the Local layer type (Same as the convolution layer but with non-shared kernel weights) Supports padding in the Local and Convolution layers (gives ability to create deep networks) Supports overlapping receptive fields in the pooling layers Supports weightless pooling layers Supports all the commonly used activation functions Supports Cross Entropy objective function in combination with a SoftMax activation function in the output layer
http://www.codepr oject.com/Articles/140631/Convolutional-Neural-Network- MNIST-Workbench
5/8
9/7/2014
Convolutional Neural Network Workbench - CodeProject Ability to specify a density percentage to generate mappings between layers far more easily Improved DataProvider class with much reduced memory footprint and a common logical functionality shared across all datasets (easier to add datasets) Much improved UI speed and general functionality
License This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)
About the Author
Filip D'haene Software Developer Belgium No Biography provided Article Top
Comments and Discussions Search this forum
Add a Comment or Question Profile popups Spacing Relaxed
Noise Medium
Layout Normal
Go
Per page 25
Update
First Prev Next
My vote of 5
CatchExAs
Re: My vote of 5 Change database [modified]
Re: Change database Re: Change database Re: Change database Re: Change database [modified] Re: Change database Please guide me to training CNNs on Sound spectrum Images
Re: Please guide me to training CNNs on Sound spectrum Images [modified] Re: Please guide me to training CNNs on Sound spectrum Images rbf wrong
http://www.codepr oject.com/Articles/140631/Convolutional-Neural-Network- MNIST-Workbench
3hrs 40mins ago
Filip D'haene
2hrs 34mins ago
MikeMNN
13-Jun-14 7:24
Filip D'haene
13-Jun-14 16:41
MikeMNN
16-Jun-14 8:20
Filip D'haene
17-Jun-14 9:30
MikeMNN
18-Jun-14 4:09
Filip D'haene
18-Jun-14 18:08
Member 10627683
12-Apr-14 6:53
Filip D'haene
14-Apr-14 13:43
Member 10627683
14-Apr-14 21:40
wsnzone
20-Mar-14 5:47
6/8
9/7/2014
Convolutional Neural Network Workbench - CodeProject Re: rbf wrong Re: rbf wrong Zip file corrupted
Re: Zip file corrupted Please guide me to implement GPU accelerated version
Re: Please guide me to implement GPU accelerated version
Re: Why didn't CNN-CIFAR-10 use higher image resolution on input layer? how can i store the training setting and tra ining result in my database?
Re: how can i store the training setting and training result in my database? Re: how can i store the training setting and training result in my database? Question about time for each epoch
News
Question
24-Mar-14 15:25
Ido Freeman
17-Mar-14 8:27
Filip D'haene
17-Mar-14 8:43
Member 10642225
8-Mar-14 4:53
11-Mar-14 15:37
Member 10372081
2-Mar-14 1:06
Filip D'haene
6-Mar-14 17:41
Member 10334053
17-Dec-13 5:23
Filip D'haene
30-Dec-13 13:39
Member 10088555
21-Jan-14 19:47
27-Nov-13 22:59
29-Nov-13 7:49
Filip D'haene
Last Update: 9-Jul-14 8:46 Suggestion
wsnzone
tintinkool1988
Re: Question about time for each epoch
General
20-Mar-14 13:10
Filip D'haene
Why didn't CNN-CIFAR-10 use higher image resolution on input layer? [modified]
Last Visit: 31-Dec-99 23:00
Filip D'haene
Refresh Bug
Answer
Joke
1 2 3 4 5 6 7 Next »
Rant
Admin
Use Ctrl+Left/Right to switch m essages, Ctrl+Up/Down to switc h threads, Ctrl+Shift+Left/Right to switch pages.
Permalink | Advertise | Privacy | Mobile Web04 | 2.8.140705.1 | Last Updated 8 Jul 2014
Layout: fixed | fluid
http://www.codepr oject.com/Articles/140631/Convolutional-Neural-Network- MNIST-Workbench
Article Copyright 2010 by Filip D'haene Everything else Copyright © CodeProject, 1999-2014
7/8
9/7/2014
Convolutional Neural Network Workbench - CodeProject Terms of Service
http://www.codepr oject.com/Articles/140631/Convolutional-Neural-Network- MNIST-Workbench
8/8