Linear models are not always able to sufficiently capture the structure of a dataset. Sometimes, combining predictors in a non-parametric method, such as deep neural networks (DNNs), would yield a more flexible modeling of the response variables in the predictions. Furthermore, the standard statistical classification or regression approaches are inefficient when dealing with more complexity, such as a high-dimensional problem, which usually suffers from multicollinearity. For confronting these cases, penalized non-parametric methods are very useful. This paper proposes two heuristic approaches and implements new shrinkage penalized cost functions in the DNN, based on the elastic-net penalty function concept. In other words, some new methods via the development of shirnkaged penalized DNN, such as ${\mathrm{DNN}}_{\mathrm{elastic-net}}$ and ${\mathrm{DNN}}_{\mathrm{ridge}\&\mathrm{bridge}}$, are established, which are strong rivals for ${\mathrm{DNN}}_{\mathrm{Lasso}}$ and ${\mathrm{DNN}}_{\mathrm{ridge}}$. If there is any dataset grouping information in each layer of the DNN, it may be transferred using the derived penalized function of elastic-net; other penalized DNNs cannot provide this functionality. Regarding the outcomes in the tables, in the developed DNN, not only are there slight increases in the classification results, but there are also nullifying processes of some nodes in addition to a shrinkage property simultaneously in the structure of each layer. A simulated dataset was generated with the binary response variables, and the classic and heuristic shrinkage penalized DNN models were generated and tested. For comparison purposes, the DNN models were also compared to the classification tree using GUIDE and applied to a real microbiome dataset.

Generalized additive models as a predictor in regression approaches, are made up over cubic spline basis and penalized regression splines. Despite of linear predictor in GLM, generalized additive models use a sum of smooth functions of covariates as a predictor. The data which are used in this study have generalized Pareto distribution and have been simulated by inversion method. The data are generated in two types, the stationary case and the non-stationary case. The method of root mean square of errors as a method of measurement is used for comparison between power of predictions which are based on penalized regression splines as a method in univariate generalized additive models and linear regression based on maximum likelihood estimation. The finding of this research illustrates that the amount of accuracy of estimation of parameter of location in UGAM approach as an alternative promising of modelling through each specialized GPD's models, has less RMSE in compare with MLE.