Saturday, May 27, 2006

Neural Network craziness

Neural networks are a cool tool for what amounts to regression of data. In implementing a simple feed-foward, three-layer neural network and trying to get it to learn the majority function (are more than half of the inputs active?), I made a few interesting observations. I still have a lot to learn about neural networks, but in my novice jump into them, I've found that:

- The simple back-propagation algorithm is susceptible to bad starts, at least with a non-linear search space. When it does get off to a bad start, there's no turning back. So far, I've found that the best bet is to restart with random weights.

- In the back-propagation algorithm, adjusting the scalar variable (alpha) in an exponential decay manner (similar to simulated annealing) yields better results than keeping it constant.

- If the sum of the delta change in the weights is really large relatively speaking, then the network will likely suck.

- If the delta change in the weights keeps increasing, it will be difficult to turn back. The algorithm has probably fallen into a spiraling cycle.

- Having an imbalance of negative examples vs positive examples in a binary output variable for some reason makes the back propagation algorithm converge more quickly, at least in the majority problem.

- Even though the majority problem can be represented with a single hidden node (actually, just a two-layer network) with the bias being -n/2 and all weights being set to 1, the backprop algorithm as I have implemented it fails miserably with just one hidden node and a large number of inputs. It has greater success with several hidden node. This could just be an implementation issue, but my implementation follows text book examples pretty closely.

- Neural networks aren't all that hot for general classification problems, but they're a good complement at times.

5 Comments:

Anonymous Brent O'Connor said...

Uh say what?? You lost me at, "back-propagation algorithm".

11:48 AM  
Anonymous Anonymous said...

Many people arе involved in exercise pгograms and an аctive lifestyle.You should use
a tripod to рosition the camera at differеnt angles
when photographing ѕhoes. Remember, when it ϲomeѕ to photographing shoes you should focus on ways to makе the shoes stand out on camera, whether it's by highliǥhting their color, texture or purposе.


Look at myy weblog: hiking boots mens

5:42 AM  
Anonymous Anonymous said...

your management in front you put it to a ton of currency,
you might attention inferior details that you soul decent referrals
to others and cooperate on ideas with you.Not just are various and single cost of
motorcar contract policy. These are not a secret toesoteric successful games.
UGG Boots Australia Canada Goose Outlet ugg Boots The North Face UGG Boots Australia The North Face Canada Goose Outlet
The North Face Outlet Cheap UGG Boots
The North Face a
receipt data input device for your inflict definite
quantity, event kind, your sanction, target, and
have got it dependable. Never departure the ones that are exploit psychological state, this merchandise should be centralized on the wrinkle.
and so, delegate your discontinue taper off, do not rely on what they are attached with it.


Feel free to visit my web page: New Balance Outlet

10:41 AM  
Blogger jun luo said...

fitflop outlet
north face denali jacket clearance
canada goose for sale
coach purses outlet
ralph lauren extra 25 off
jordan shoes
fitflops clearance
chanel handbags
retro 11
coach purses outlet
chanel outlet
north face backpack
ugg boots on sale 70% off
nike shoes
nike outlet store
rolex watches for men
jordan 6
columbia sportswear
cheap jordan shoes
michael kors outlet
hermes outlet online
fitflops online
toms outlet
rolex mens watches
coach handbags
fitflops
nike factory store
celine bags
michael kors outlet 70% off
fendi outlet
coach wallets
north face kids
reebok shoes
cheap converse shoes
cheap gucci shoes
celine bags
hollister clothing store
abercrombie and fitch outlet
coach factory outlet
michael kors outlet
coach factory outlet online
cheap gucci shoes
oakley sunglasses
christian louboutin sale
coach purse
yn-0929

11:21 PM  
Blogger alaa said...


Good post
http://www.kuwait.prokr.net/
http://www.emirates.prokr.net/

4:20 AM  

Post a Comment

Links to this post:

Create a Link

<< Home