Sesi 09 Dosen Pembina : Danang Junaedi IF-UTAMA 1 IF-UTAMA 2

Transkripsi

1 esi 09 Dosen Pembina : Danang Junaedi IF-UTAMA 1 IF-UTAMA Given a collection of records (training set ) Each record contains a set of attributes, one of the attributes is the class. Find a model for class attribute as a function of the values of other attributes. Goal: previously unseen records should be assigned a class as accurately as possible. A test set is used to determine the accuracy of the model. Usually, the given data set is divided into training and test sets, with training set used to build the model and test set used to validate it. IF-UTAMA 3 IF-UTAMA 4 1

2 10 10 Tid Attrib1 Attrib Attrib3 Class 1 Large 15K Medium 100K 3 mall 70K 4 Medium 10K 5 Large 95K 6 Medium 60K 7 Large 0K 8 mall 85K 9 Medium 75K 10 mall 90K Tid Attrib1 Attrib Attrib3 Class 11 mall 55K? 1 Medium 80K? 13 Large 110K? 14 mall 95K? 15 Large 67K? Learn Model Apply Model Predicting tumor cells as benign or malignant Classifying credit card transactions as legitimate or fraudulent Classifying secondary structures of protein as alpha-helix, beta-sheet, or random coil Categorizing news stories as finance, weather, entertainment, sports, etc IF-UTAMA 5 IF-UTAMA 6 Decision Tree based Methods Rule-based Methods Memory based reasoning Neural Networks Naïve Bayes and Bayesian Belief Networks upport Vector Machines Mengubah data menjadi pohon keputusan (decision tree) dan aturan-aturan keputusan (rule) IF-UTAMA 7 IF-UTAMA 8

3 A decision tree is a chronoical representation of the decision problem. Each decision tree has two types of nodes; round nodes correspond to the states of nature while square nodes correspond to the decision alternatives. The branches leaving each round node represent the different states of nature while the branches leaving each square node represent the different decision alternatives. At the end of each limb of a tree are the payoffs attained from the series of branches making up that limb. IF-UTAMA 9 IF-UTAMA 10 Diagnosa penyakit tertentu, seperti hipertensi, kanker, stroke dan lain-lain Pemilihan produk seperti rumah, kendaraan, komputer dan lain-lain Pemilihan pegawai teladan sesuai dengan kriteria tertentu Deteksi gangguan pada komputer atau jaringan komputer seperti Deteksi Entrusi, deteksi virus (trojan dan varians) dll IF-UTAMA 11 IF-UTAMA 1 3

4 Tid Marital tatus 1 ingle 15K Married 100K 3 ingle 70K 4 Married 10K Cheat 5 Divorced 95K 6 Married 60K 7 Divorced 0K 8 ingle 85K 9 Married 75K 10 ingle 90K Training Data plitting Attributes Mart ingle, Divorced Married TaxInc < 80K > 80K YE Model: Decision Tree Tid Marital tatus 1 ingle 15K Married 100K 3 ingle 70K 4 Married 10K Cheat 5 Divorced 95K 6 Married 60K 7 Divorced 0K 8 ingle 85K 9 Married 75K 10 ingle 90K Married Mart ingle, Divorced TaxInc < 80K > 80K There could be more than one tree that fits the same data! YE IF-UTAMA 13 IF-UTAMA 14 Tid Attrib1 Attrib Attrib3 Class 1 Large 15K Medium 100K 3 mall 70K 4 Medium 10K 5 Large 95K 6 Medium 60K 7 Large 0K 8 mall 85K 9 Medium 75K Learn Model 10 mall 90K Tid Attrib1 Attrib Attrib3 Class 11 mall 55K? 1 Medium 80K? 13 Large 110K? 14 mall 95K? 15 Large 67K? Apply Model Decision Tree IF-UTAMA 15 IF-UTAMA 16 4

5 tart from the root of tree. Test Data Marital tatus Cheat Test Data Marital tatus Cheat Married 80K? Married 80K? Mart Mart ingle, Divorced Married ingle, Divorced Married TaxInc TaxInc < 80K > 80K < 80K > 80K YE YE IF-UTAMA 17 IF-UTAMA 18 Test Data Test Data Marital tatus Cheat Marital tatus Cheat Married 80K? Married 80K? Mart Mart ingle, Divorced Married ingle, Divorced Married TaxInc TaxInc < 80K > 80K < 80K > 80K YE YE IF-UTAMA 19 IF-UTAMA 0 5

6 Test Data Test Data Marital tatus Cheat Marital tatus Cheat Married 80K? Married 80K? Mart ingle, Divorced Married Mart ingle, Divorced Married Assign Cheat to TaxInc < 80K > 80K TaxInc < 80K > 80K YE YE IF-UTAMA 1 IF-UTAMA Tid Attrib1 Attrib Attrib3 Class 1 Large 15K Medium 100K 3 mall 70K 4 Medium 10K 5 Large 95K 6 Medium 60K 7 Large 0K 8 mall 85K 9 Medium 75K 10 mall 90K Learn Model Tid Attrib1 Attrib Attrib3 Class 11 mall 55K? 1 Medium 80K? 13 Large 110K? 14 mall 95K? 15 Large 67K? Apply Model Decision Tree IF-UTAMA 3 IF-UTAMA 4 6

7 IF-UTAMA 5 IF-UTAMA 6 Data dinyatakan dalam bentuk tabel dengan atribut dan record. Atribut menyatakan suatu parameter yang dibuat sebagai kriteria dalam pembentukan tree. Misalkan untuk menentukan main tenis, kriteria yang diperhatikan adalah cuaca, angin dan temperatur. alah satu atribut merupakan atribut yang menyatakan data solusi per-item data yang disebut dengan target atribut. Atribut memiliki nilai-nilai yang dinamakan dengan instance. Misalkan atribut cuaca mempunyai instance berupa cerah, berawan dan hujan. IF-UTAMA 7 IF-UTAMA 8 7

8 1. Mengubah bentuk data (tabel) menjadi model tree. ID3 Algorithm C.45 Algorithm etc. Mengubah model tree menjadi rule Disjunction (v OR) Conjunction (^ AND) 3. Menyederhanakan Rule (Pruning) IF-UTAMA 9 IF-UTAMA 30 IF-UTAMA 31 IF-UTAMA 3 8

9 IF-UTAMA 33 IF-UTAMA 34 IF-UTAMA 35 IF-UTAMA 36 9

10 Given a set of examples,, categorised in categories ci, then: 1. Choose the root node to be the attribute, A, which scores the highest for information gain relative to.. For each value v that A can possibly take, draw a branch from the node. 3. For each branch from A corresponding to value v, calculate v. Then: If v is empty, choose the category cdefault which contains the most examples from, and put this as the leaf node category which ends that branch. If v contains only examples from a category c, then put c as the leaf node category which ends that branch. Otherwise, remove A from the set of attributes which can be put into nodes. Then put a new node in the decision tree, where the new attribute being tested in the node is the one which scores highest for information gain relative to v (note: not relative to ). This new node starts the cycle again (from ), with replaced by v in the calculations and the tree gets built iteratively like this. The algorithm terminates either when all the attributes have been exhausted, or the decision tree perfectly classifies the examples. IF-UTAMA 37 IF-UTAMA 38 pesifikasikan masalah menentukan Atribut dan Target Atribut berdasarkan data yang ada Hitung nilai Entropy dari setiap kriteria dengan data sample yang ditentukan. Hitung Information Gain dari setiap kriteria de terpilih adalah kriteria dengan Information Gain yang paling tinggi. Ulangi sampai diperoleh node terakhir yang berisi target atribut Entropy() adalah jumlah bit yang diperkirakan dibutuhkan untuk dapat mengekstrak suatu kelas (+ atau -) dari sejumlah data acak pada ruang sample. Entropy bisa dikatakan sebagai kebutuhan bit untuk menyatakan suatu kelas. emakin kecil nilai Entropy maka semakin baik untuk digunakan dalam mengekstraksi suatu kelas. Panjang kode untuk menyatakan informasi secara optimal adalah p bits untuk messages yang mempunyai probabilitas p. IF-UTAMA 39 IF-UTAMA 40 10

11 IF-UTAMA 41 IF-UTAMA 4 Training Data et IF-UTAMA 43 IF-UTAMA 44 11

12 IF-UTAMA 45 IF-UTAMA 46 IF-UTAMA 47 IF-UTAMA 48 1

13 IF-UTAMA 49 IF-UTAMA 50 IF-UTAMA 51 IF-UTAMA 5 13

14 ample Atribut Target Atribut Decision Tree?? IF-UTAMA 53 IF-UTAMA 54 Jumlah instance = 8 Jumlah instance positif = 3 Jumlah instance negatif = 5 Entropy ( Hipertensi ) = = = = ( ) ( ) ( ) ( ) = 0, ,44 = 0,955 Jumlah instance = 8 Instance Usia Muda Instance positif = 1 Instance negatif = 3 Tua Instance positif = Instance negatif = Entropy Usia Entropy ( Muda ) = P ins ( ) Entropy Tua = Entropy(muda) = Entropy(tua) = 1 P ins P ins P ins IF-UTAMA 55 IF-UTAMA 56 14

15 Gain v (, Usia ) Entropy ( ) Entropy ( ) = Entropy 4 = 8 = = 0.00 = Muda Tua ( ) Entropy ( ) Entropy ( ) 4 8 ( 0.955) ( ) ( 1) v Muda, Tua Muda v Tua Jumlah instance = 8 Intance Berat Overweight Instance positif = 3 Instance negatif = 1 Average Instance positif = 0 Instance negatif = Underweight Instance positif = 0 Instance negatif = Entropy Entropy Entropy ( Overweight ) = ( Average ) = ( Underweigh t) = Entropy(Overweight)=0.906 Entropy(Average)=0.5 Entropy(Underweight)=0.5 IF-UTAMA 57 IF-UTAMA 58 Gain v (, Berat ) Entropy ( ) Entropy ( ) = Entropy 4 = = Overweight Average t ( ) Entropy ( ) Entropy ( ) Underweigh Entropy ( ) ( ) ( ) ( 0.5) ( 0.5) = 0,5 = v Overwight, Average, Underweigh t Overweight v average Underweigh t Jumlah instance = 8 Intance Jenis Kelamin Pria Instance positif = Instance negatif = 4 Wanita Instance positif = 1 Instance negatif = 1 Entropy ( Pr ia ) = Entropy ( Wanita ) = Entropy(Pria)=1 Entropy(Wanita)=0.75 IF-UTAMA 59 IF-UTAMA 60 15

16 Gain v (, JenisKela min ) Entropy ( ) Entropy ( ) = Entropy 6 = 8 8 = = 0,017 Pr ia Wanita ( ) Entropy ( ) Entropy ( ) ( ) ( 1) ( 0.75) = Pr ia v Pr ia, Wanita Wanita v Atribut yang dipilih adalah atribut berat karena nilai Information Gainnya paling tinggi Overweight Berat Average Underweight Jumlah Instance untuk Overweight = 4 Jumlah Instance untuk Average = Jumlah Instance untuk Underweight = Hitung Gain paling tinggi untuk dijadikan cabang berikutnya IF-UTAMA 61 IF-UTAMA 6 Jumlah instance = 4 Instance (Berat = Overwight ) & Usia = Muda Instance positif = 1 Instance negatif = 0 Tua Instance positif = Instance negatif = 1 Instance (Berat = Overwight ) & Jenis Kelamin = Pria Instance positif = Instance negatif = 1 Wanita Instance positif = 1 Instance negatif = 0 Clasification Rule???? 63 IF-UTAMA 64 16

17 Underfitting and Overfitting Underfitting: when model is too simple, both training and test errors are large Overfitting results in decision trees that are more complex than necessary Missing Values Costs of Classification Pre-Pruning (Early topping Rule) top the algorithm before it becomes a fully-grown tree Typical stopping conditions for a node: top if all instances belong to the same class top if all the attribute values are the same More restrictive conditions: top if number of instances is less than some user-specified threshold top if class distribution of instances are independent of the available features (e.g., using χ test) top if expanding the current node does not improve impurity measures (e.g., Gini or information gain). IF-UTAMA 65 IF-UTAMA 66 Post-pruning Grow decision tree to its entirety Trim the nodes of the decision tree in a bottom-up fashion If generalization error improves after trimming, replace sub-tree by a leaf node. Class label of leaf node is determined from majority class of instances in the sub-tree Can use MDL for post-pruning Class = 0 Class = 10 Error = 10/30 A1 A Training Error (Before splitting) = 10/30 Pessimistic error = ( )/30 = 10.5/30 Training Error (After splitting) = 9/30 Pessimistic error (After splitting) = ( )/30 = 11/30 A? PRUNE! A4 A3 Class = 8 Class = 3 Class = 4 Class = 5 Class = 4 Class = 4 Class = 1 Class = 1 IF-UTAMA 67 IF-UTAMA 68 17

18 Missing values affect decision tree construction in three different ways: Affects how impurity measures are computed Affects how to distribute instance with missing value to child nodes Affects how a test instance with missing value is classified Tid Marital tatus 1 ingle 15K Married 100K 3 ingle 70K 4 Married 10K IF-UTAMA 69 IF-UTAMA 70 Class 5 Divorced 95K 6 Married 60K 7 Divorced 0K 8 ingle 85K 9 Married 75K 10? ingle 90K Missing value Before plitting: Entropy(Parent) = -0.3 (0.3)-(0.7)(0.7) = plit on : Class Class = = = 0 3 = 4 =? 1 0 Entropy(=) = 0 Entropy(=) = -(/6)(/6) (4/6)(4/6) = Entropy(Children) = 0.3 (0) (0.9183) = Gain = 0.9 ( ) = Tid Marital tatus Class 1 ingle 15K Married 100K 3 ingle 70K 4 Married 10K 5 Divorced 95K 6 Married 60K 7 Divorced 0K 8 ingle 85K 9 Married 75K Class= 0 Class= 3 Cheat= Cheat= 4 Tid Marital tatus Class= 0 + 3/9 Class= 3 Class 10? ingle 90K Class= + 6/9 Class= 4 Probability that = is 3/9 Probability that = is 6/9 Assign record to the left child with weight = 3/9 and to the right child with weight = 6/9 New record: Tid Marital tatus ingle, Divorced TaxInc 11? 85K? Mart < 80K > 80K YE Class Married Married ingle Divorced Total Class= Class= 6/ Total Probability that Marital tatus = Married is 3.67/6.67 Probability that Marital tatus ={ingle,divorced} is 3/6.67 IF-UTAMA 71 IF-UTAMA 7 18

19 IF-UTAMA 73 IF-UTAMA 74 IF-UTAMA 75 IF-UTAMA 76 19

20 1. Dr. Mourad YKHLEF.009. Decision Tree Induction ystem.king aud University. Achmad Basuki, Iwan yarif Decision Tree. Politeknik Elektronika Negeri urabaya (PEN) IT 3. imon Colton.004. Decision Tree Learning.- 4. Tom M. Mitchell Machine Learning. Mc- Graw Hill IF-UTAMA 77 0