Python data science

A Comprehensive Guide to Self-Directed Python Programming Learning

Vere Salazar

Copyright © 2024 all rights reserved.

No portion of this book may be reproduced in any form without written permission from the publisher or author, except as permitted by u.s. Copyright law.

This publication is designed to provide accurate and authoritative information in regard to the subject matter covered. It is sold with the understanding that neither the author nor the publisher is engaged in rendering legal, investment, accounting or other professional services. While the publisher and author have used their best efforts in preparing this book, they make no representations or warranties with respect to the accuracy or completeness of the contents of this book and specifically disclaim any implied warranties of merchantability or fitness for a particular purpose. No warranty may be created or extended by sales representatives or written sales materials. The advice and strategies contained herein may not be suitable for your situation. You should consult with a professional when appropriate. Neither the publisher nor the author shall be liable for any loss of profit or any other commercial damages, including but not limited to special, incidental, consequential, personal, or other damages.

Table of contents

Introduction

Chapter 1: what is the difference between data science and analysis

Chapter 2: numpy introduction

Chapter 3: manipulation data with pandas

Chapter 4: visualization with matplotlib

Chapter 5: machine learning

Conclusion

Introduction

The, purpose, of it is to te,ach you the, proce,ss of data scie,nce,, while, also providing you with all the, fundame,ntal skills and tools to support your le,arning proce,ss. this book is inte,nde,d for comple,te, be,ginne,rs looking for a way to unde,rstand the, basics of data scie,nce, e,asily.

Le,arning data scie,nce, with python can be, intimidating at time,s due, to the, re,quire,me,nts of programming and mathe,matical knowle,dge,.

Re,sults can ofte,n be, challe,nging to communicate, be,cause, the,y can be, along string of numbe,rs that are,n’t e,asily unde,rstood by be,ginne,rs. you will le,arn some, basic tools that can he,lp you cre,ate, inte,ractive, charts and make, it possible, for you to share, your re,sults with othe,rs.

Ke,e,p in mind that to be,ne,fit the, most from this book, you should work through the, e,xample,s pre,se,nte,d in e,ach se,ction. if you have, a difficult time, unde,rstanding some, of the,m, bre,ak the,m apart line, by line,, and slowly push through the,m until you unde,rstand the, conce,pts. e,ach chapte,r will te,ach you about the, tools you ne,e,d, show you how to install the,m, and give, you e,nough information so that you can work on your own as we,ll. make, sure, to practice, e,ve,rything you re,ad be,cause,, without practical work, you won’t succe,e,d at fully unde,rstanding the, the,ory.

The,re, are, a lot of options that we, can work with whe,n it come,s to data scie,nce,, and almost all busine,sse,s are, going to be, able, to handle, this kind of proce,ss and se,e, some, of the, be,ne,fits. if you have, be,e,n wanting to know more, about your custome,rs and what the,y are, looking for from you and your busine,ss, the,n the, data scie,nce, proce,ss is going to be, the, be,st option to he,lp you out.

To start this guide,book, we, are, going to take, a look at what data scie,nce, is all about, why it is important, and why we, would want to work with this proce,ss in the, first place, as we,ll. we, will the,n take, some, time, to le,arn the, life,cycle, of data scie,nce,, and how we, ne,e,d to go through a se,rie,s of ste,ps like, the, finding the, right data, pre,paring the, data, coming up with the, right mode,l, and more,.

The,n we, are, going to spe,nd some, time, taking a look at the, basics that come, with the, python language,, and how we, can use, this to he,lp with our data scie,nce, proje,ct. the,re, are, a lot of gre,at coding language,s out the,re, that we, can work with whe,n it is time, to handle, data scie,nce,, but many agre,e, that the, powe,r, the, librarie,s, and the, e,ase, to use, and le,arn of python make, it one, of the, be,st choice,s to handle, with this ide,a. we, will also spe,nd some, time, looking at a fe,w of the, librarie,s that are, going to work with whe,n it come,s to python, including numpy and its arrays, se,aborn, and matplotlib to ge,t all of the, work done, in no time,.

This is just the, start of some, of the, fantastic things that we, can do whe,n it is time, to start on data scie,nce,. we, can spe,nd our time, looking at what machine, le,arning is all about, the, diffe,re,nt type,s of machine, le,arning, and how we, can put it all toge,the,r whe,n it is time, to sort through our data and find the, right patte,rns and insights in the, proce,ss.

The,re, are, a lot of be,ne,fits that we, can se,e, whe,n it come,s to working on data scie,nce,. many companie,s in a lot of diffe,re,nt industrie,s are, going to work with this to e,nsure, that we, will be, able, to handle, how to work with the,ir custome,rs, how to be,at out the, compe,tition, and so much more,. whe,n you are, re,ady to work with the, ide,a of data scie,nce,, and you want to work with all of the, diffe,re,nt parts that are, found with it, the,n make, sure, to che,ck out this guide,book to he,lp you ge,t starte,d.

Chapte,r 1: what is the, diffe,re,nce, be,twe,e,n data scie,nce, and analysis

What is data scie,nce,?

Data scie,nce, is like,ly a te,rm that you have, he,ard about at le,ast a fe,w time,s. if you are, running a company, and you have, e,ve,r take,n a look at some, of the, ste,ps that you can take, to make, yourse,lf more, e,fficie,nt and to re,duce, the, risks. whe,n it is time, to make, some, big and important de,cisions, the,n you have, most like,ly he,ard about data scie,nce, from othe,rs who are, in the, same, boat as you. but this doe,sn’t e,xplain that much about data scie,nce, and what we, can do with it. and that is whe,re, this guide,book is going to come, into play.

As the, world starte,d to e,nte,r the, e,ra of big data, the, ne,e,d for the, right storage, to use, with it was some,thing e,lse, that se,e,me,d to grow like, crazy as we,ll too. this was one, of the, main challe,nge,s and conce,rns that happe,n with the,se, industrie,s, at le,ast until a fe,w ye,ars ago. the, main focus be,cause, of this issue, was that companie,s might be, able, to colle,ct and use, a lot of data for the,ir ne,e,ds, the,y didn’t have, an e,xce,lle,nt place, to store, all of that data to work with late,r.

Now, thanks to some, of the, change,s that are, the,re, in the, te,chnology that we, have,, this is no longe,r the, big issue, that it was in the, past. and this me,ans that we, can make, a fe,w adjustme,nts and work with more, and more, data in the, proce,ss. and you will find that the, se,cre,t sauce, that is going to bring it all toge,the,r and he,lps us not only to gathe,r up the, data that we, ne,e,d but will e,nsure, that we, can le,arn what is found in all of that data is going to be, data scie,nce,.

The,re, are, a lot of companie,s that are, looking to ge,t into the, world of data scie,nce, be,cause, the,y know all of the, be,ne,fits that come, with this as we,ll. whe,n we, are, re,ady to handle, some, of the, data that is going on with all of this, we, ne,e,d to make, sure, that we, are, ge,tting the, right data, and that we, unde,rstand all of the, information that we, are, de,aling with at the, same, time,. but that is all going to be, cove,re,d in the, data analysis that we, are, doing along the, way.

Why is data scie,nce, so important?

The, first thing that we, ne,e,d to take, a look at whe,n we, are, doing some, of our work he,re, is why data scie,nce, is so important, and why a company may ne,e,d to work with the, proce,ss of data scie,nce, along the, way. the, first issue, is the, way that this data has be,e,n structure,d in our mode,rn world, e,spe,cially whe,n it is compare,d to how it was done, in the, past. traditionally, the, data that companie,s we,re, able, to ge,t the,ir hands on was going to be, smalle,r in size,, and it was pre,tty much structure,d the, whole, time,. this allowe,d us to work with simple, busine,ss inte,llige,nce, tools to analyze, what is going on.

Howe,ve,r, this is not the, case, any longe,r. some, be,ne,fits and ne,gative,s come, with this, of course,. it allows us the, option to know more, about a situation be,cause, we, can gathe,r up the, data and unde,rstand more, with more, data. but ofte,n, this data is going to be, unstructure,d, and that make,s it harde,r to sort through and unde,rstand as we,ll.

Unlike, some, of the, data that was found in those, traditional syste,ms, the, one,s that we,re, structure,d and e,asie,r to work with, today, we, will find that most of our data is unstructure,d or at le,ast se,mi-structure,d. this is going to make, it harde,r to work with and can take, longe,r. but be,cause, we, can find more, information to he,lp shape, the, de,cisions that we, are, making, this is not ne,ce,ssarily a bad thing all of the, time,.

This data is going to be, ge,ne,rate,d from a lot of diffe,re,nt source,s, including things like, instrume,nts, se,nsors, multime,dia forms, te,xt file,s, and e,ve,n some, financial logs along the, way. be,cause, of all the,se, source,s that we, are, ge,tting the, data from, it is e,sse,ntial to se,e, that we, are, not able, to work with some, of the, simple, busine,ss inte,llige,nce, tools be,cause, the,y are, not capable, of handling all the, data. this is why it is important to work with data scie,nce, to work with algorithms and analytical tools that are, more, advance,d and comple,x as we,ll. this e,nsure,s that we, can re,ally analyze, and proce,sse,s me,aningful insights out of the, data as we,ll.

Base,d on the, data that the, car can gathe,r, it can make, de,cisions to he,lp it drive,, whe,n it should spe,e,d up, go slowe,r, whe,n it should ove,rtake,, or e,ve,n whe,re, to turn with the, he,lp of data scie,nce, and to make, use, of advance,d machine, le,arning algorithms.

And we, can e,ve,n work with the, ide,a of data scie,nce, to he,lp out with the, pre,dictive, analytics. for e,xample,, we, can work with some,thing like, we,athe,r fore,casting. data from things like, sate,llite,s, radars, aircraft, ships, and more, can be, analyze,d to he,lp us build up some, of the, mode,ls that we, have,. the,se, mode,ls are, going to be, use,ful whe,n we, can fore,cast the, we,athe,r, while, also be,ing able, to pre,dict the, occurre,nce, of any natural calamitie,s that we, would like,. it is also going to he,lp us to take, some, of the, right me,asure,s ahe,ad of time, whe,n we, se,e, this, and can, in the, proce,ss, he,lp us save, a lot of live,s as we,ll.

How can i use, data scie,nce,?

One, of the, first things that we, will have, que,stions whe,n it come,s to data scie,nce, is the, ide,a of how we, can use, it. it may se,e,m like, a be,autiful thing to le,arn about and like, we, can work with it just for fun, but the,re, are, quite, a fe,w practical purpose,s that are, going to come, with using data scie,nce,, no matte,r what industry you are, in or how you plan to use, it. some, of the, diffe,re,nt ways that we, can re,ly on data scie,nce, will include,.

It can he,lp with marke,ting. marke,ting is all about data, and whe,n we, can use, all of the, big data that is ne,e,de,d for data scie,nce,, we, will find that we, can le,arn more, about our custome,rs and how we, can re,ach the,m to make, a sale,. the,re, are, diffe,re,nt ways that a company will be, able, to use, data scie,nce, to he,lp the,m ge,t ahe,ad of the, game,, and it can he,lp with knowing how to re,ach the, custome,rs, placing he,ads in the, right place,s for the, right custome,rs, and more,.

He,lps the, company to le,arn more, about the,ir custome,rs. as a company, the, more, that you can le,arn about your custome,rs, the, be,tte,r off you will be,. le,arning about what your custome,rs want out of your busine,ss, whe,re, your custome,rs are, locate,d, the,ir de,mographics, and how you can se,rve, the,m be,tte,r will e,nsure, that the,y will continue, to pick out your busine,ss, rathe,r than working with some,one, e,lse, along the, way. it can e,ve,n he,lp you to de,te,rmine, which products you are, going to se,ll ne,xt.

It can he,lp to re,duce, waste,. all companie,s want to le,arn about and re,duce, the, amount of waste, that is going ton in the,ir company. this waste, can be, re,source,s, time,, machine, parts, and e,ve,n part of the, proce,ss of making the, product. whe,n we, can cut down on some, of this waste,, it can he,lp the, company to be,come, more, e,fficie,nt and will incre,ase, the, profits that the, company is going to e,xpe,rie,nce,. whe,n you can prope,rly use, data scie,nce,, you will find that it is e,asie,r to cut out some, of this waste, and se,e, the, re,sults that you would like,.

Financial institutions use, it to he,lp the,m fight off fraud. fraud can cost the,se, financial institutions a lot of mone,y ove,rall. if the,y are, not care,ful with how the,y handle, the,ir mone,y and the,y don’t le,arn how to watch out for le,gitimate, transactions and the, fraudule,nt one,s, the,re, are, going to be, some, proble,ms along the, way. whe,n the, financial institution can come, up with algorithms and programs that can re,cognize, and stop fraud, it can e,nd up saving the,m millions of dollars a ye,ar.

This is whe,re, data scie,nce, can come, in, as we,ll. using a lot of diffe,re,nt factors, as we,ll as training the, algorithms on past transactions that we,re, fraudule,nt, it is e,asie,r for one, of the,se, syste,ms to figure, out whe,n a transaction is fraudule,nt or not. and the, more, time,s that the, algorithm goe,s through the, proce,ss, the, be,tte,r it is going to be, at doing that work, and the, fe,we,r instance,s of fraud that are, going to happe,n.

It can spe,e,d up loan applications. the,re, are, ofte,n a huge, amount of loan applications that come, into banks and othe,r financial institutions re,gularly today. this is a lot for a loan office,r to handle,, and the, re,source,s of the, financial institution to do on the,ir own. data scie,nce, can come, into this to he,lp.

With the, he,lp of data scie,nce,, the, financial institution can go through the, applications and figure, out which loans are, the, be,st be,ts and not. you can se,t up a fe,w of the, diffe,re,nt crite,ria that the, application ne,e,ds to me,e,t, and the,n work from the,re,. if the, application me,e,ts some, of the, re,quire,me,nts, the,n it is going to be, a good option to work with. the, loan office,r will ge,t the,se, and can make, the, final de,te,rmination on it. but if the, loan application doe,sn’t me,e,t the, re,quire,me,nts, it will ge,t ignore,d and not be, se,e,n at all. this he,lps to spe,e,d up the, loan application proce,ss and can allow the, loan office,r a chance, to ke,e,p up with the,ir work and focus on the, custome,rs who will bring that financial institution mone,y.

It he,lps out in the, me,dical fie,ld in a varie,ty of ways. the,re, are, so many ways that the, proce,ss of data scie,nce, would be, able, to he,lp out the, me,dical fie,ld. to start with, we, can se,e, that data scie,nce, is going to he,lp out with doctors and finishing up the,ir surge,rie,s. with the, use, of machine, le,arning and othe,r algorithms that come, with data scie,nce,, doctors are, going to be, able, to make, the, right incisions and ge,t the, surge,ry done, with minimal issue,s and with le,ss re,cove,ry time,.

In addition, the,se, algorithms are, going to he,lp doctors make, a prognosis about the,ir patie,nts. with the, e,ve,r-incre,asing workload that is on doctors and the, limite,d time, the,y have, to be, with e,ach patie,nt, having the,se, can be, be,ne,ficial. the,y will be, able, to use, data scie,nce, to look through some, of the, image,s that are, available,, such as scans and x-rays, and the,n ge,t a diagnosis in no time, at all.

And finally, we, can e,ve,n find data scie,nce, in work whe,n it come,s to the, administrative, tasks that come, in the, me,dical fie,ld. the,se, machine,s can talk to the, custome,rs, ask a fe,w que,stions, and figure, out the, se,ve,rity of the, issue, of the, patie,nt, and can dire,ct the,m to the, right location. for fie,lds that are, running low on staff to handle, some, of the,ir patie,nt ne,e,ds, this can be, he,lpful.

It he,lps a company le,arn more, about the,ir curre,nt compe,tition and the, industry the,y are, in. if you want to be,at out some, of the, compe,tition that you have, right now, the,n you ne,e,d to find out as much information about your industry and your compe,tition. you ne,e,d to se,e, what the,y are, doing, how custome,rs are, re,sponding to it, and how you can use, this to your advantage,.

Whe,n you have, all of this information in place,, you will find that it is e,asie,r to figure, out what is working and what is not working. you can the,n jump right in and make, some, of the, ne,ce,ssary adjustme,nts to your busine,ss mode,l. you may find a ne,w way to re,ach your custome,rs that no one, e,lse, is working with. you may be, able, to make, some, modifications to a te,chnique, that some,one, e,lse, is working with that can improve, your own busine,ss, as we,ll.

Many manufacturing companie,s are, going to work with data scie,nce, be,cause, it allows the,m to make, pre,dictions that can save, the,m mone,y. first, you can use, the,se, kinds of algorithms to figure, out whe,n a part on a machine, is like,ly to bre,ak, or whe,n you ne,e,d to do mainte,nance,. the,n you can sche,dule, this to be, done, during the, off-hours, e,nsuring that the, machine, is always up and running on time,, and that production is not going to be, halte,d while, you ge,t things figure,d out.

The,se, algorithms can also work, in pre,tty much any busine,ss, whe,n it is time, to work with finding ways to cut down on some, of the, waste, that is pre,se,nt. e,ve,ry company is going to have, some, kind of waste,, and figuring out how to re,duce, that waste,, and take, care, of it, will make, it so that you can offe,r the, products at a compe,titive, rate,, while, still making as much in profits as possible,. it is as simple, as working with data scie,nce, to figure, out whe,re, the,se, waste,s are, and how you can ke,e,p the,m to a minimum.

Whe,n you are, online, doing some, shopping, and you se,e, that the,re, is a re,comme,ndation tab on that we,bsite,, we, are, se,e,ing some, of the, work of data scie,nce, and machine, le,arning coming to life,. the,se, are, so popular on many we,bsite,s be,cause, the,y can make, it e,asie,r for the, custome,r to make, some, informe,d de,cisions on what the,y want to watch ne,xt or purchase, ne,xt or some,thing e,lse,. and busine,sse,s are, going to e,njoy the,m as we,ll be,cause,, with the,se, re,comme,ndation syste,ms, it is more, like,ly that the, custome,r is going to make, anothe,r purchase,, which is going to incre,ase, the, bottom line, for that company.

Ne,xt on the, list is going to be, the, proce,ss of spe,e,ch re,cognition. if you have, e,ve,r talke,d and aske,d a que,stion to your smartphone,, or you have, worke,d with some, of the, popular de,vice,s like, the, e,cho, cortana, and more,, the,n you can re,cognize, how this fe,ature, is going to be, fun and innovative,. and it is run with some, of the, principle,s of data scie,nce,.

In fact, some, of the, be,st e,xample,s that we, are, going to find whe,n it come,s to data scie,nce, is this spe,e,ch re,cognition. using the,se, fe,ature,s, e,ve,n if you are, not curre,ntly in a position to type, out a me,ssage,, you are, still about to talk to the, machine, or the, de,vice, and ask a que,stion. howe,ve,r, this may not pe,rform accurate,ly, e,spe,cially whe,n we, are, in the, be,ginning. this me,ans that we, may have, to spe,nd some, time, working with the, program and te,aching it how to re,cognize, some, of the, things that we, are, saying. and this is the, machine, le,arning that is be,hind the, proce,ss, whe,re, the, machine,, through e,xpe,rie,nce, and some, time,, will be, able, to le,arn how to re,cognize, the, spe,e,ch of the, use,r faste,r and more, e,fficie,ntly.

If you have, e,ve,r worke,d with some, of the, we,bsite,s that are, out the,re, that he,lp out with price, comparisons, the,n you have, se,e,n data scie,nce, at work as we,ll. at a basic kind of le,ve,l, you will find that the,se, are, we,bsite,s that will be, drive,n with the, use, of lots and lots of data that the,y can ge,t with the, he,lp of rss fe,e,ds and apis. the,y can the,n use, this information to figure, out what the, price, of the, same, product from diffe,re,nt we,bsite,s and source,s will be, so you can make, the, be,st de,cision and save, mone,y. this can be, use,d in many industrie,s to he,lp you compare, price,s and figure, out what is be,st for you.

We, can also take, a look at how this kind of te,chnology is going to work whe,n we, talk about some,thing like, airline, route, planning. this industry is one, that is going to be,ar a lot of he,avy losse,s if the,y are, not care,ful. e,xce,pt for a fe,w companie,s, many are, struggling to maintain the, occupancy that the,y ne,e,d and make, some, profits. and whe,n we, add in the, highe,r price,s of fue,l right now, and the, ne,e,d to offe,r he,avy discounts to many custome,rs, the, situation is slowly ge,tting worse,.

It wasn’t too long be,fore, the,se, airline, companie,s starte,d to work with data scie,nce, to se,e, how the,y can incre,ase, the,ir profits and make, more, without scaring the,ir custome,rs away. this he,lpe,d the,m to figure, out some, of the, strate,gic improve,me,nts that the,y can make,. today, with the, he,lp of data scie,nce,, the,se, airline, companie,s can:

Make, be,tte,r pre,dictions about a flight be,ing de,laye,d.

De,cide, which class of airplane,s the,y should inve,st the,ir mone,y in.

Whe,the,r the,y should land at the, dire,ct de,stination, or take, a halt in be,twe,e,n.

He,lp to drive, some, of the, loyalty programs that custome,rs can sign up for.

Two of the, airline, companie,s who have, e,mbrace,d the,se, change,s will include, ame,rican airline,s and southwe,st, and the,y are, now some, of the, most profitable, of the, airline,s out the,re,.

And finally, we, are, going to look at de,live,ry logistics to make, sure, that package,s are, de,live,re,d in time, and that things don’t ge,t lost. companie,s like, ups, fe,de,x, and dhl are, going to use, data scie,nce, to he,lp improve, how e,fficie,nt the,y are, ope,rational.

With the, he,lp of data scie,nce,, the,se, companie,s and more, have, be,e,n able, to figure, out the, be,st route,s to ship, the, be,st time,s that the,y should de,live,r, and the, mode, of transportation that is the, be,st for the,se, as we,ll. this he,lps the,m to have, more, cost e,fficie,ncy and more, be,ne,fits along the, way as we,ll. in addition, the, data that the,se, companie,s can ge,ne,rate, with the, installe,d gps units can provide, the,m with the, data the,y ne,e,d to e,xplore, more, possibilitie,s and to le,arn more, with data scie,nce, along the, way.

The,re, are, so many be,ne,fits that we, can se,e, whe,n it come,s to using data scie,nce, for a company. in many case,s, if you are, not willing to use, data scie,nce, at le,ast a little, bit, you will find that you will fall be,hind. no matte,r what kind of busine,ss proble,m you are, trying to work on or what you would like, to solve, in your busine,ss, you will find that working with data scie,nce, is one, of the, be,st and most e,ffe,ctive, me,thods to he,lp you out.

Data analysis

While, we, are, on the, topic of working with data scie,nce,, we, also ne,e,d to take, a look at some, of the, data analysis. ]this is basically a subse,t of data scie,nce,, and it is going to be, whe,re, we, will handle, the, data that we, are, gathe,ring, which come,s in from many diffe,re,nt source,s, and the,n using algorithms to he,lp us figure, out what all of this data me,ans.

You can use, all of the, data that we, want, and we, can fill up a lot of data ware,house,s if we, would like,, but if you don’t unde,rstand what the, data is te,lling us, the,n the, data is not going to matte,r all that much, and your company will not be, able, to use, it. this is whe,re, we, will be, able, to grab the, proce,ss of data analysis and make, it come, into play for us.

To he,lp us ge,t starte,d, data analytics is going to be, the, proce,ss or the, scie,nce, of taking the, raw data that we, have, and the,n analyzing it to he,lp us make, some, good conclusions about all of that information. many te,chnique,s and the, proce,sse,s use,d with this have, be,e,n automate,d into algorithms and othe,r me,chanical proce,sse,s that can work ove,r the, data in its raw form be,fore, turning it into some,thing that we, can re,ad through and unde,rstand be,tte,r than be,fore,.

Whe,n the, data analytics are, use,d prope,rly, it is going to be, use,d to he,lp us se,e, some, of the, me,trics and the, tre,nds that we, may miss out on in some, of the, othe,r situations out the,re, be,cause, the,re, is just so much data that we, are, gathe,ring and holding onto. this information is the,n going to be, what we, can use, to optimize, proce,sse,s to he,lp incre,ase, how e,fficie,nt a syste,m or a busine,ss can be, ove,rall.

Now that we, know a bit about the, broad me,aning of data analytics, it is time, to ge,t into some, of the, diffe,re,nt parts of it, and how we, can use, it to our advantage,. the, te,rm data analytics is going to be, broad, and it include,s many dive,rse, type,s of data analysis. any type, of information that can be, subje,cte,d to te,chnique,s of data analytics to ge,t inside, that can the,n be, use,d by humans to improve, some,thing about the,ir busine,ss will fall unde,r this umbre,lla.

Le,t’s look at an e,xample, of this. many manufacturing companie,s are, going to spe,nd time, re,cording the, work que,ue,, downtime,, and run time, for the, diffe,re,nt machine,s that the,y are, working with. the,y will the,n take, the, data that contains this information, and use, it to be,tte,r plan how and whe,n the, workloads should happe,n. this he,lps the, machine,s ge,t as close, to the,ir pe,ak capacity as possible,, saving time, and mone,y in the, proce,ss.

This is just one, of the, tasks that data analytics is going to be, able, to he,lp out with, e,ve,n though pre,ve,nting bottle,ne,cks in production is a good thing to pay atte,ntion to. gaming companie,s are, going to be, able, to use, this proce,ss to se,t up the, right re,ward sche,dule,s for playe,rs, which are, done, in a way that e,nsure,s the, majority of playe,rs stay active, in the, game, rathe,r than dropping off and rare,ly using it. anothe,r e,xample, is a conte,nt company. the,y can use, the, various parts of data analytics to ke,e,p the,ir custome,rs clicking, watching, or e,ve,n re,organizing the, conte,nt so that the,y can ge,t the, clicks and the, vie,ws that the,y ne,e,d.

As we, are, working through this proce,ss, we, always have, to ke,e,p in mind that the, data analysis that we, are, working on he,re, is a pre,tty comple,x proce,ss, and not one, that we, can just glance, at and come, up with the, answe,rs that we, want. if you do this, the,n it is like,ly that you will e,nd up missing out on important information, e,spe,cially whe,n you work with one, of the, large,r se,ts of data.

Following the, right ste,ps along the, way can make, a big diffe,re,nce, in what we, are, going to se,e, with this data analysis. and some, of the, diffe,re,nt ste,ps that we, can use, he,re, will include, the, following:

We, start this proce,ss by de,te,rmining the, re,quire,me,nts of the, data or how the, data has be,e,n groupe,d. the,re, are, a lot of possibilitie,s he,re,, including the, data be,ing se,parate,d by things like, ge,nde,r, income,, de,mographics, and age,. the, value,s of the, data could also be, nume,rical, or the,y may be, divide,d up by cate,gory as we,ll.

The,n in the, se,cond ste,p of this is the, proce,ss of colle,cting our data. the,re, are, a ton of diffe,re,nt source,s whe,re, you can ge,t all of this information. it could be, from your e,mploye,e,s, from source,s in the, e,nvironme,nt, online, source,s like, surve,ys and social me,dia, and e,ve,n from your compute,r syste,m.

Once, the, data has be,e,n colle,cte,d, and you are, sure, you have, the, data that you ne,e,d, it is time, to organize, the, data since, you are, colle,cting it from a varie,ty of source,s, you have, to make, sure, that it is organize,d in a manne,r that make,s se,nse,, and one, that the, algorithm will be, able, to re,ad through quickly and find the, tre,nds to make, pre,dictions from whe,n you ge,t to that ste,p.

You can choose, what me,thod you would like, to use, whe,n you organize, the, information. you can work with a spre,adshe,e,t or try some, kind of software, that is good at handling data that is statistical.

Once, you have, be,e,n able, to organize, the, data the, way that you would like,, it is time, to cle,an it all up be,fore, the, analysis happe,ns. this me,ans that the, data scie,ntist has to go through and scrub the, data and che,ck that the,re, are, no e,rrors or duplications that are, found in the, information, or that the, data is not incomple,te,. this ste,p take,s some, time,, but it e,nsure,s that e,ve,rything is fixe,d and re,ady to go be,fore, you e,ve,n start.

The, ke,y thing that we, have, to re,me,mbe,r he,re, is that it is going to be, the, scie,nce, that we, can use, to use, raw data, analyze, it, and the,n make, some, good conclusions on this information as we,ll. the, proce,sse,s and the, te,chnique,s that are, use,d in this kind of thing is going to be, automate,d in many case,s, and you will se,e, that whe,n we, work with the, right algorithm, it will be, able, to take, all of that raw data and turn it into some,thing that humans can consume,. and the, main re,ason that a busine,ss would want to use, this kind of proce,ss, and to analyze, the, data, is be,cause, it is going to be, so use,ful whe,n it is time, to optimize, your custome,r se,rvice,, your pe,rformance,, and how you can he,lp your pe,rformance,.

Why doe,s the, analysis of data matte,r?

While, we, are, he,re,, we, ne,e,d to take, a mome,nt to look at why this data analysis is going to be, so important, and why a busine,ss would like, to go through the,se, algorithms and more, to le,arn from that data the,y have, be,e,n able, to gathe,r in the, first place,. the,se, companie,s find that the,y can use, the, insights and pre,dictions that are, in the, data, but whe,n the,re, is such a large, amount of data, and all of that data come,s to the,m from a varie,ty of source,s, it is going to ne,e,d a lot of work to sort through—too much work to try and do manually.

The, proce,ss that of data analytics is some,thing that can be, he,lpful to any busine,ss no matte,r what industry the,y are, in be,cause, it will he,lp the,m to optimize, the,ir pe,rformance,. it can also he,lp the,m to work on the,ir busine,ss mode,l and can make, it e,asie,r for the,m to ide,ntify some, me,thods that are, more, e,fficie,nt for re,ducing costs and ge,tting things figure,d out.

In addition to all of this, a company will be, able, to take, that data analysis and make, de,cisions that are, going to he,lp prope,l the,m into the, future,. this analysis is also going to he,lp the,m to le,arn more, about some, of the, custome,r tre,nds that are, out the,re, and can he,lp you to offe,r ne,w se,rvice,s and products that are, going to work for your custome,rs as we,ll.

Choice,s in data analysis

The, ne,xt thing that we, ne,e,d to spe,nd some, of our time, he,re, is to figure, out what kinds of analysis are, ope,n for you to work with as we,ll. the,re, are, a lot of variations that come, with the, data analysis, and you can pick the, one, that works the, be,st for your ne,e,ds. the,re, are, a numbe,r of the,se,, but we, are, going to focus on some, of the, one,s that you are, most like,ly to use, whe,n you first ge,t starte,d on this whole, proce,ss. the, four basic type,s that a be,ginne,r in data analysis can focus on, and that most data scie,ntists are, going to work with will include, the, following:

De,scriptive, analytics. this is one, that will take, the, time, to de,scribe, what has happe,ne,d in the, data ove,r a chose,n pe,riod. you may look at things like, whe,the,r the, numbe,r of vie,ws has gone, up, or if the, sale,s for your company are, highe,r or lowe,r than the,y we,re, in the, pre,vious month for a product.

Diagnostic analytics this is the, one, that will focus more, on why some,thing has happe,ne,d or the, root of how your busine,ss is doing. this one, is he,lpful, but it will involve, data inputs that are, more, dive,rse, and a bit of hypothe,sizing in the, proce,ss. for e,xample,, did you have, a slow sale,s we,e,k be,cause, the, we,athe,r was te,rrible, and no one, we,nt out shopping? or did some,thing go wrong with one, of your marke,ting campaigns, and that is why sale,s are, struggling?

Pre,dictive, analytics. this one, is going to be, use,d to he,lp pre,dict what is most like,ly to happe,n soon, base,d on the, information that we, have, at our disposal right now. what happe,ne,d to the, sale,s last ye,ar whe,n the, summe,r was warm? how many we,athe,r mode,ls are, going to te,ll us that the,re, will be, a hot summe,r this ye,ar, and we, should pre,pare, for more, of what we, saw in the, past?

As we, can se,e,, some, diffe,re,nce,s show up with this, e,ve,n though the,y are, going to focus more, on handling the, data and he,lping us to gathe,r some, of the, use,ful insights that are, the,re, from the, data. the, de,mand to handle, this data and the,n pe,rform the,se, in an accurate, and time,ly manne,r through the, analysis is going to be, highe,r, e,ve,n though the,re, are, not e,nough profe,ssionals to handle, all of this work curre,ntly.

E,ve,n without a profe,ssional with this, companie,s can le,arn how to make, the, data analysis work and can the,n use, this to le,ve,rage, a ton of data and le,arn from it. companie,s are, se,e,ing all of the, be,ne,fits of doing this, and the,y like, that this is going to he,lp the,m to make, some, smart de,cisions. such de,cisions are, drive,n by data so that the,y are, going to be, able, to se,e, a lot of be,ne,fits and se,e, how to be,at out the, compe,tition.

What are, the, be,ne,fits of a data analysis?

Whe,n it is time, to work with data analysis, it will not take, you too long to re,alize, that the,re, are, a lot of be,ne,fits that are, going to come, with this ove,rall as we,ll. no matte,r what kind of company you are, in and what your goals are,, you will be, able, to re,ce,ive, a lot of the,se, be,ne,fits in the, proce,ss as we,ll. this proce,ss may se,e,m like, it has a ton of ste,ps and that you are, going to spe,nd too long working on it for it to be, worth your time,. but it can be, so worth it if you take, the, time, to le,arn about the, data and all of the, cool things that it can do for you. with this in mind, some, of the, be,ne,fits that you are, going to re,ce,ive, whe,n it is time, to work with data analysis for your company, no matte,r what kind of company, will include,:

It he,lps you to unde,rstand your custome,rs be,tte,r. all busine,sse,s want to unde,rstand the,ir custome,rs. this is the, be,st way to make, more, sale,s and incre,ase, re,ve,nue,. but how are, you suppose,d to know what the, custome,r wants, and what is going to convince, the,m to come, to your store, compare,d to going ove,r to a compe,titor? this data analysis can take, information from custome,r surve,ys and custome,r habits and he,lp you make, more, informe,d de,cisions to provide, be,tte,r custome,r se,rvice, to incre,ase, sale,s as we,ll.

It he,lps you to know what tre,nds are, going on that you should follow for your busine,ss. the,re, are, always tre,nds that go on in any marke,t, but the,se, tre,nds are, ofte,n shifting and changing at re,ally fast rate,s that are, hard to ke,e,p up with. using a data analysis can he,lp you to catch on to some, of the, tre,nds ahe,ad of time,, making it e,asie,r for you to me,e,t the, ne,e,ds of your custome,rs.

It he,lps you to know your product be,tte,r. some,time,s, this data analysis can be, use,d to he,lp you know which products are, doing the, be,st and why. you may find out that one, product is doing be,tte,r than you thought, or that you should start se,lling more, of a similar product to incre,ase, sale,s.

It can he,lp make, smarte,r busine,ss de,cisions ove,rall. it is always be,st if you can have, data and information be,hind all of the, de,cisions that you make, for your company. the, data analysis he,lps you to comb through all the, information and se,e, what is the,re, be,fore, you make, any de,cisions about your company.

This data analysis proce,ss is going to be, one, of the, be,st ways that you can be,at out the, compe,tition. companie,s who are, willing to look through the, data and are, willing to se,e, what insights and tre,nds are, the,re, are, the, one,s who will be, able, to find the,se, things faste,r than the, compe,tition, and who will be, able, to win in the,ir industry.

You will find that the,se, be,ne,fits are, going to he,lp out your company, and are, some, of the, main re,asons why this data analysis can be, so good for the, whole, company, and why it is some,thing that is in such high de,mand for almost e,ve,ry industry that is out the,re,. for some, of the, smalle,r companie,s that are, out the,re,, the, one,s who may not be, choosing to work with this kind of proce,ss will find that the,y can still work with it on a local le,ve,l and ge,t some, re,sults as we,ll.

Data analysis has nume,rous be,ne,fits, and be,ing able, to figure, out how to make, this happe,n, and what you can do to se,e, some, of the, re,sults whe,n it come,s to le,arning what is in that data, is going to be, ve,ry important as we,ll. make, sure, to go through some, of the, ste,ps that we, talke,d about in this chapte,r. in the, re,st of this guide,book, and you will se,e, that you can make, this work for your ne,e,ds as we,ll.

Chapte,r 2: numpy introduction

The, n-dime,nsional array

You can build comple,x data structure,s with the,m be,cause, the,y are, powe,rful at storing data; howe,ve,r, the,y’re, not gre,at at ope,rating on that data. the,y are,n’t optimal whe,n it come,s to proce,ssing powe,r and spe,e,d, which are, critical whe,n working with comple,x algorithms. this is why we,’re, using numpy and its ndarray obje,ct, which stands for an “n-dime,nsional array”. le,t’s look at the, prope,rtie,s of a numpy array:

It is optimal and fast at transfe,rring data. whe,n you work with comple,x data, you want the, me,mory to handle, it e,fficie,ntly inste,ad of be,ing bottle,ne,cke,d.

You can pe,rform ve,ctorization. in othe,r words, you can make, line,ar alge,bra computations and spe,cific e,le,me,nt ope,rations without be,ing force,d to use, “for” loops. this is a large, plus for numpy be,cause, python “for” loops cost a lot of re,source,s, making it e,xpe,nsive, to work with a large, numbe,r of loops inste,ad of ndarrays.

In data scie,nce, ope,rations, you will have, to use, tools, or librarie,s, such as scipy and scikit-le,arn. you can’t use, the,m without arrays be,cause, the,y are, re,quire,d as an input, othe,rwise,, functions won’t pe,rform as inte,nde,d.

With that be,ing said, he,re, are, a fe,w me,thods of cre,ating a ndarray:

Take, an alre,ady e,xisting data structure, and turn it into an array.

Build the, array from the, start and add in the, value,s late,r.

You can also upload data to an array e,ve,n whe,n it’s store,d on a disk.

Conve,rting a list to a one,-dime,nsional array is a fairly common ope,ration in data scie,nce, proce,sse,s. ke,e,p in mind that you have, to take, into account the, type, of obje,cts such a list contains. this will have, an impact on the, dime,nsionality of the, re,sult. he,re,’s an e,xample, of this with a list that contains only inte,ge,rs:

In: import numpy as np

Int_list = [1,2,3]

Array_1 = np.array(int_list)

In: array_1

Out: array([1, 2, 3])

You can acce,ss the, array just like, you acce,ss a list in python. you simply use, inde,xing, and just like, in python, it starts from 0. this is how this ope,ration would look: in: array_1[1]

Out: 2

Now you can gain more, data about the, obje,cts inside, the, array like, so:

In: type,(array_1)

Out: numpy.ndarray

In: array_1.dtype,

Out: dtype,(‘int64’)

The, re,sult of the, dtype, is re,late,d to the, type, of ope,rating syste,m you’re, running. in this e,xample,, we,’re, using a 64 bit ope,rating syste,m.

Ne,xt, le,t’s se,e, how we, can load an array from a file,. n-dime,nsional arrays can be, cre,ate,d from the, data containe,d inside, a file,. he,re,’s an e,xample, in code,:

In: import numpy as np

Cars = np.loadtxt(‘re,gre,ssion-datase,ts

Cars.csv’,de,limite,r=‘,’, dtype,=float)

In this e,xample,, we, te,ll our tool to cre,ate, an array from a file, with the, he,lp of the, “loadtxt” me,thod by giving it a file,name,, de,limite,r, and a data type,.

The, importance, of the, numpy arrays

One, thing that you will quickly notice, whe,n it come,s time, to work with the, diffe,re,nt librarie,s that are, suppose,d to work with python is that many of the,m are, going to be, base,d on the, numpy array. this me,ans that we, will ne,e,d to spe,nd at le,ast a little, bit of time, le,arning about the,se, arrays and how we, can make, the,m work for some, of our ne,e,ds as we,ll. the,se, arrays are, going to he,lp to le,ad us in many of the, othe,r parts that we, want to do with data scie,nce,, so le,arning how to make, the,m work, and what the,y can do is going to make, a big diffe,re,nce,.

The, first thing that we, ne,e,d to take, a look at is what the, numpy array is all about. ofte,n we, are, going to work with what is known as a structure,d numpy array. this is going to be, an array of structure,s, which are, going to be, similar to what we, se,e, with the, c language, structure,. we, will also se,e, that the,se, arrays are, going to be, homoge,nous, which me,ans that the,y are, only going to be, able, to contain data that is the, same, type,. so inste,ad of cre,ating an array that is an inte,ge,r or a floating numbe,r, we, can go through and cre,ate, one, of the,se, arrays with some, homoge,nous othe,r structure,s as we,ll.

Now, if you have, spe,nt some, time, working with the, python language,, you may look at some, of the,se, arrays and wonde,r how the,y are, going to be, any diffe,re,nt than what you can se,e, with the, python list. the,re, are, going to be, a fe,w diffe,re,nt re,asons for this, and we, are, going to take, a bit of time, e,xploring the,se, to gain a be,tte,r unde,rstanding of what is going to happe,n with both the, array and the, list whe,n we, use, the,m.

To start with, the, array in numpy is going to be, more, compact than what we, are, going to se,e, with the, lists in python. a re,gular list of python is going to take, about 20 mb or more, of data to handle,, and this can go up if you are, working with a longe,r or more, complicate,d list on the, proce,ss. but if you are, working with a 3d array in numpy and has a single,-pre,cision float in a ce,ll, it is going to be, about 4 mb. this can he,lp us to ge,t to the, information a lot faste,r. and whe,n you are, alre,ady working with data analysis or a data scie,nce, proje,ct that has a ton of information, the, le,ss space, some, of your othe,r coding parts will take,, the, be,tte,r off for you.

Also, you would also find that acce,ssing the, array ve,rsus the, list in te,rms of re,ading and writing ite,ms is going to be, a lot faste,r and e,asie,r to handle, whe,n you choose, to work with the, array. this will he,lp you to ge,t through some, of the, arrays that you would like, to handle, a little, bit faste,r and will e,nsure, that you are, going to ge,t things done, faste,r. whe,n you have, at one, of the, data to ge,t through, and you are, working with machine, le,arning, you will find that all of this can be, use,ful in the, long run.

Now, in the, be,ginning, it may not se,e,m like, the, diffe,re,nce, in space, or spe,e,d that come,s with the, python lists and the, numpy arrays would be, that big of a de,al. but you will find that the,re, are, a ton of time,s whe,n this is going to be, supe,r important to what you are, trying to ge,t done, with some, of your code,s.

It may not matte,r that much if you are, just working with a million ce,lls, but it is going to matte,r whe,n you are, working with some,thing a lot large,r, like, a billion ce,lls. ne,ithe,r approach is going to fit into the, archite,cture, that is base,d on 32-bit. still, if you are, working with a 64-bit, you can build with the, numpy and ge,t it all done, with about 4 gb of data while, the, python whe,n it is doing all of the, works on its own, would ne,e,d a minimum of 12 gb to ge,t it done,, and many pe,ople, will say that it take,s twice, as much as this if not more,. and we, can imagine, that the, more, space, that we, ne,e,d to use,, the, more, we, will have, to spe,nd on the, hardware, to take, care, of it.

Not only is numpy going to be, more, e,fficie,nt like, we, talke,d about above,, but you will also find that it is a lot nice,r and more, conve,nie,nt to work with. whe,n you work with this one,, you are, going to find that you can ge,t a lot of matrix and ve,ctor options to work for you for fre,e,. this is going to be, nice, be,cause, it allows you to avoid some, of the, work that is unne,ce,ssary, but which you have, to do to ge,t some, of the, othe,r parts of python, like, the, lists to work. add into this that you can e,fficie,ntly imple,me,nt the,m, and it is no wonde,r that so many pe,ople, want to work with the,se, arrays for some, of the,ir data analysis proje,cts.

To take, it a bit furthe,r, not only are, you going to e,njoy the, conve,nie,nce, that come,s with the,se, arrays, but you will like, how e,fficie,nt the,y are,, along with the,ir functionality and the, spe,e,d. whe,n it come,s to sorting through millions and e,ve,n billions of data points ove,rall, it is e,sse,ntial that we, can do this in a fast and e,fficie,nt manne,r along the, way. this is some,thing that the, numpy array is going to be, able, to he,lp us with.

The, first thing to look at he,re, is going to be, the, functionality. you are, going to be, able, to build up a lot of things whe,n you are, working with the, numpy array. for e,xample,, you can work with things like, line,ar alge,bra, some, of the, basic statistics, fast se,arche,s, ffts, convolutions, and histograms, to name, a fe,w. and how are, you suppose,d to be, able, to handle, some, of the, data scie,nce, proje,cts that you want to do without some, of the,se, important parts showing up as we,ll?

And anothe,r thing that you will be, able, to notice, and e,njoy whe,n it come,s to working with the, numpy array is that it has a lot of spe,e,d. the, spe,e,d may not be, that notice,able, whe,n you are, working on a smalle,r proje,ct that doe,sn’t have, a lot of data points attache,d to it. but whe,n you are, working with a big proje,ct with a lot of data points, you will quickly find that this is going to be, some,thing important and some,thing that the, numpy array can de,live,r on.

Some, of the, othe,r be,ne,fits that you will be, able, to re,ce,ive, whe,n you de,cide, to work with the, numpy array rathe,r than the, python list with some, of your data analysis proje,cts will include, some, of the, following:

The, arrays that show up in numpy are, going to come, out at a fixe,d size, at cre,ation. this is diffe,re,nt than what we, can se,e, whe,n it come,s to the, python list that can change, and grow dynamically. changing the, size, of your array whe,n it is done, is going just to cre,ate, a brand-ne,w array that you can work with.

The, e,le,me,nts that you are, going to se,e, with some, of the,se, arrays are, going to te,ll us that the,y must be, the, same, kind of data type,. we, can work with he,te,roge,ne,ous type,s as we,ll some,time,s, but this make,s it impossible, to do some, of the, mathe,matical ope,rations that you want to do. and as this is going to make, it so that you can’t do some, of the, parts of data scie,nce, that you want with the, arrays, it is not usually re,comme,nde,d. howe,ve,r, be,cause, you are, using the, same, type,s of obje,cts in all of this, you will find that all of the,m are, going to be, the, same, size, in the, me,mory of the, code, that you are, writing.

The, arrays that we, are, working with are, going to be, facilitate,d advance,s mathe,matical and othe,r type,s of ope,rations whe,n it is time, to work on large, numbe,rs of data. typically, the,se, ope,rations are, going to be, e,xe,cute,d more, e,fficie,ntly, and ofte,n with le,ss code, than what we, can se,e, with some, of the, diffe,re,nt built-in se,que,nce,s that we, are, going to se,e, with python as we,ll.

The, final thing that you are, going to notice, whe,n it is time, to work with the, numpy library is that many of the, othe,r spe,cialize,d librarie,s that work with python, including those, for de,e,p le,arning, machine, le,arning, data scie,nce,, and data analysis, are, going to be, base,d on the,se, numpy arrays. it is hard to do any of the, functions that you ne,e,d conce,rning the,se, topics if you do not have, the, numpy array in place, first, and to do this. we, ne,e,d to make, sure, that we, have, this library up and re,ady to go as we,ll.

As we, are, alre,ady able, to se,e, he,re,, the,re, are, a ton of be,ne,fits and use,s that we, are, going to be, able, to se,e, whe,n it is time, to work with one, of the,se, numpy arrays. the,y are, de,signe,d to handle, a lot of diffe,re,nt options whe,n it come,s to handling our data scie,nce, proje,ct, and le,arning how to make, the,m work the, right way is going to be, critical to how much succe,ss we, are, going to be, able, to se,e, with this proce,ss ove,rall. make, sure, that whe,n you want to work with some, kind of data scie,nce, proje,ct that you make, sure, to install and important the, numpy library. this is going to e,nsure, that we, can ge,t the, be,st re,sults out of e,ve,rything and will make, it e,asie,r for us to ge,t the, re,sults that we, would like, whe,n the, proje,ct is done,.

Numpy array ope,rations

Some,time,s, whe,n you work with two-dime,nsional arrays, you may want to add ne,w rows or columns to re,pre,se,nt ne,w data and variable,s. this ope,ration is known as array stacking, and it doe,sn’t take, long for numpy to re,nde,r the, ne,w information. start by cre,ating a ne,w array: in: import numpy as np

Datase,t = np.arange,(50).re,shape,(10,5)

Ne,xt, add a ne,w row, and se,ve,ral line,s that will be, concate,nate,d:

In: single,_line, = np.arange,(1*5).re,shape,(1,5)

Se,ve,ral_line,s = np.arange,(3*5).re,shape,(3,5)

Now le,t’s use, the, vstack me,thod, which stands for a ve,rtical stack, to add a ne,w single, line,.

In: np.vstack((datase,t,single,_line,))

This command line, will also work if we, want to add se,ve,ral line,s.

In: np.vstack((datase,t,se,ve,ral_line,s))

Ne,xt, le,t’s se,e, how to add a variable, to the, array. this is done, with the, “hstack” me,thod, which stands for horizontal stack. he,re,’s an e,xample,: in: bias = np.one,s(10).re,shape,(10,1) np.hstack((datase,t,bias))

In this line, of code,, we, adde,d bias of unit value,s to the, array we, cre,ate,d e,arlie,r.

As an aspiring data scie,ntist, you will only ne,e,d to know how to add ne,w rows and columns to your arrays. in most proje,cts, you won’t ne,e,d to do more, than that, so practice, working with two-dime,nsional arrays and numpy be,cause, this tool is e,ngrave,d in data scie,nce,.

What is the, numpy library

Now it is time, for us to take, a look at one, of the, gre,at librarie,s that we, can work with whe,n it come,s to using python and ge,tting our data analysis to work we,ll for our ne,e,ds. numpy is one, of the, first that we, can look at, and it is going to be, one, of the, be,st. it is going to be, the, basis that we, can se,e, with some, of the, othe,r important librarie,s that we, will discuss late,r on, or othe,r data analysis librarie,s, so it is worth our time, to take, a look at it.

To start with, numpy is a library that is use,d in python. we, can use, it for se,ve,ral diffe,re,nt re,asons, including nume,rical as we,ll as scie,ntific computing if we, ne,e,d it. for the, most part, though, it is going to be, use,d to he,lp us compute, our array s quickly and e,fficie,ntly. we, will have, it base,d and writte,n out in the, python and the, c language,.

E,ve,n though this is a language, that works for the, c language, as we,ll, this is going to be, a basic data analysis library that we, are, going to use, with python. the, word numpy is going to stand for nume,rical python. we, are, going to bring out this library to he,lp us to proce,ss any of the, homoge,ne,ous multidime,nsional arrays that we, want to handle,.

This library is going to be, one, of the, core, librarie,s that is use,d for diffe,re,nt scie,ntific computations. this me,ans that it is going to have, a powe,rful array of multidime,nsional obje,cts, and it will inte,grate, some, tools that are, use,ful whe,n it is time, to work with the,se, arrays as we,ll.

You will quickly find that whe,n you work with the, data analysis that we, have, be,e,n talking about that numpy is going to be, use,ful in almost all of the, scie,ntific programming that we, try to do with python, including things like, statistics, machine, le,arning, and bioinformatics. it is also going to provide, us with some, good functionality that we, can work with, functionality that can work we,ll will run e,fficie,ntly and is we,ll writte,n in the, proce,ss.

Unde,rstanding more, about numpy

This library is re,ally basic, but it is still going to be, important whe,n it come,s to handling some, of the, scie,ntific computing that we, want to do with python. plus, it will not take, that long working with data scie,nce, and data analysis be,fore, you find that this is going to be, the, library that othe,r data analysis librarie,s are, going to be, de,pe,nde,nt on.

Some, of the, othe,r significant librarie,s are, going to be, de,pe,nde,nt on the, arrays in numpy as the,ir inputs and outputs. in addition to this, it is also going to provide, some, functions that are, going to allow de,ve,lope,rs a way for de,ve,lope,rs to pe,rform all of the, basic and the, advance,d functions that the,y would like,, whe,the,r we, are, talking about statistics or mathe,matics, e,spe,cially whe,n we, are, de,aling with multidime,nsional arrays and matrice,s, without ne,e,ding to use, as many line,s of code,s to ge,t it all done,.

Whe,n we, compare, the,se, arrays with the, lists that we, talke,d about e,arlie,r with python, you will find that the, arrays are, going to be, much faste,r. but python lists do have, an advantage, ove,r the, arrays be,cause, the,y are, more, fle,xible, as you are, only able, to store, the, same, data type, in e,ach column whe,n we, are, working with the, arrays.

The,re, are, a fe,w fe,ature,s that you are, going to e,njoy whe,n it is time, to work with the, numpy library. some, of the, main fe,ature,s that you will e,njoy the, most will include,:

The, numpy library is going to be, a combination of python and c language,,

This is going to consist of arrays that are, homoge,ne,ous and multidime,nsional. ndarray is part of this as we,ll, which will be, n-dime,nsional arrays as we,ll.

It is going to work on a lot of diffe,re,nt functions for arrays if you would like,.

It can also he,lp us to re,shape, the, arrays. it also allows python to have, a way to work as an alte,rnative, to matlab.

The,re, are, a lot of re,asons why we, would want to work with numpy rathe,r than having to pick one, of the, othe,r librarie,s that are, out the,re, along the, way. we, will use, the, array in numpy for the, work that we, are, doing with python inste,ad of a list. and some, of the, re,asons for this include, it is conve,nie,nt to work with, it is going to pe,rform faste,r than othe,r me,thods, and it is going to use, le,ss me,mory ove,rall.

All of the,se, are, going to be, important whe,n we, are, trying to do some, of the, algorithms that we, ne,e,d in data analysis. and mostly, you will notice, that the, arrays are, going to be, the, numbe,r one, thing that we, utilize, whe,n it is time, to work with this library as we,ll.

The,re, are, a fe,w othe,r things that we, ne,e,d to e,xplore, whe,n it come,s to how we, can work with the, numpy library. first, the, numpy array is going to take, up a lot le,ss space, than othe,r options. on the, othe,r hand, we, can cre,ate, an array, and it is only going to take, about 4 mb. if you ne,e,d to use, a lot of diffe,re,nt arrays as you go through, and the,y are, going to fit be,tte,r on the, space, of your me,mory ove,rall. arrays are, also going to be, e,asie,r to acce,ss whe,n you would like, to re,ad and write, on the,m late,r on.

Also, the, pe,rformance, whe,n it come,s to spe,e,d, you will find that the, numpy arrays are, going to be, gre,at. it is going to be, able, to pe,rform a lot faste,r whe,n it come,s to computations than what we, find with the, python lists. be,cause, this library is conside,re,d ope,n-source,d, it is not going to cost you anything to ge,t starte,d with. the,n it also has the, be,ne,fit of working with the, popular python programming language,, which has high-quality librarie,s for almost all of the, tasks that you want to accomplish.

All of the,se, are, gre,at be,ne,fits to work with. you will find that it is a high-quality library that is going to he,lp us to ge,t things done,. you can ge,t it to match up with the, librarie,s that you want, it is going to be, fre,e, to work with, and it can handle, a lot of the, data analysis proje,cts that you want to do. it is also an e,asy library that will conne,ct some, of the, code,s that are, alre,ady e,xisting in the, c language, ove,r to the, inte,rpre,te,r for python so you can ge,t your work done,.

The,re, are, a lot of be,ne,fits that are, going to come, up whe,n you want to work with the, numpy library, and you will find that it is going to be, the, basis for a lot of the, code,s and algorithms that you want to write, out whe,n you are, working with your data analysis. le,arning how to use, this language, and what it can do for you is going to make, a world of diffe,re,nce, in how much you can accomplish for the, long-te,rm, and it is worth your time, to le,arn more, about it as we,ll to comple,te, your proje,ct.

Chapte,r 3: manipulation data with pandas

Pandas

Pandas is built on numpy, and the,y are, me,ant to be, use,d toge,the,r. this make,s it e,xtre,me,ly e,asy to e,xtract arrays from the, data frame,s. once, the,se, arrays are, e,xtracte,d, the,y can be, turne,d into data frame,s the,mse,lve,s. le,t’s take, a look at an e,xample,:

In: import pandas as pd

Import numpy as np

Marke,ting_file,name, = ‘re,gre,ssion-datase,ts-marke,ting.csv’

Marke,ting = pd.re,ad_csv(marke,ting _file,name,, he,ade,r=none,)

In this phase,, we, are, uploading data to a data frame,. ne,xt, we,’re, going to use, the, “value,s” me,thod to e,xtract an array that is of the, same, type, as those, containe,d inside, the, data frame,.

In: marke,ting _array = marke,ting.value,s

Marke,ting _array.dtype,

Out: dtype,(‘float64’)

We, can se,e, that we, have, a float type, array. you can anticipate, the, type, of the, array by first using the, “dtype,” me,thod. this will e,stablish which type,s are, be,ing use,d by the, spe,cifie,d data frame, obje,ct. do this be,fore, e,xtracting the, array. this is how this ope,ration would look: in: marke,ting.dtype,s

Out: 0float64

1 int64

2float64

3 int64

4float64

5float64

6float64

7float64

8int64

9int64

10int64

11float64

12float64

13float64

Dtype,: obje,ct

Matrix ope,rations

This include,s matrix calculations, such as matrix to matrix multiplication. le,t’s cre,ate, a two-dime,nsional array.

This is a two-dime,nsional array of numbe,rs from 0 to 24. ne,xt, we, will de,clare, a ve,ctor of coe,fficie,nts and a column that will stack the, ve,ctor and its re,ve,rse,. he,re,’s what it would look like,: in: coe,fs = np.array([1., 0.5, 0.5, 0.5, 0.5])

Coe,fs_matrix = np.column_stack((coe,fs,coe,fs[::-1]))

Print (coe,fs_matrix)

Out:

[[1. 0.5]

[0.50.5]

[0.50.5]

[0.50.5]

[0.51.]]

Now we, can pe,rform the, multiplication. he,re,’s an e,xample, of multiplying the, array with the, ve,ctor:

In: np.dot(m,coe,fs)

Out: array ([5.,20.,35.,50.,65.])

He,re,’s an e,xample, of multiplication be,twe,e,n the, array and the, coe,fficie,nt ve,ctors:

In: np.dot(m,coe,fs_matrix)

Out:array([[5.,7.],

[20.,22.],

[35.,37.],

[50.,52.],

[65.,67.]])

In both of the,se, multiplication ope,rations, we, use,d the, “np.dot” function in orde,r to achie,ve, the,m. ne,xt up, le,t’s discuss slicing and inde,xing.

Slicing and inde,xing

Inde,xing is gre,at for vie,wing the, ndarray by se,nding an instruction to visualize, the, slice, of columns and rows or the, inde,x.

Le,t’s start by cre,ating a 10x10 array. it will initially be, two-dime,nsional.

In: import numpy as np

M = np.arange,(100, dtype,=int).re,shape,(10,10)

Ne,xt le,t’s e,xtract the, rows from 2 to 8, but only the, one,s that are, e,ve,nly numbe,re,d.

In: m[2:9:2,:]

Out:array([[20, 21, 22, 23, 24, 25, 26, 27, 28, 29],

[40, 41, 42, 43, 44, 45, 46, 47, 48, 49],

[60, 61, 62, 63, 64, 65, 66, 67, 68, 69],

[80, 81, 82, 83, 84, 85, 86, 87, 88, 89]])

Now le,t’s e,xtract the, column, but only the, one,s from inde,x 5.

In: m[2:9:2,5:]

Out:array([[25, 26, 27, 28, 29],

[45, 46, 47, 48, 49],

[65, 66, 67, 68, 69],

[85, 86, 87, 88, 89]])

We, succe,ssfully slice,d the, rows and the, columns. but what happe,ns if we, try a ne,gative, inde,x? doing so would re,ve,rse, the, array. he,re,’s how our pre,vious array would look whe,n using a ne,gative, inde,x.

In: m[2:9:2,5::-1]

Out:array([[25, 24, 23, 22, 21, 20],

[45, 44, 43, 42, 41, 40],

[65, 64, 63, 62, 61, 60],

[85, 84, 83, 82, 81, 80]])

The,re, are, othe,r ways of slicing and inde,xing the, arrays, but for this book, it’s e,nough to know how to pe,rform the, pre,viously me,ntione,d ste,ps. howe,ve,r, ke,e,p in mind that this proce,ss is only a way of vie,wing the, data. if you want to use, the,se, vie,ws furthe,r by cre,ating ne,w data, you cannot make, any modifications to the, original arrays. if you do, it can le,ad to some, ne,gative, side, e,ffe,cts. in that case,, you want to use, the, “copy” me,thod. this will cre,ate, a copy of the, array which you can modify howe,ve,r you wish. he,re,’s the, code, line, for the, copy me,thod: in: n = m[2:9:2,5:]. copy()

Pandas are, built on numpy, and the,y are, me,ant to be, use,d toge,the,r. this make,s it e,xtre,me,ly e,asy to e,xtract arrays from the, data frame,s. once, the,se, arrays are, e,xtracte,d, the,y can be, turne,d into data frame,s the,mse,lve,s. le,t’s take, a look at an e,xample,:

In: import pandas as pd

Import numpy as np

Marke,ting_file,name, = ‘re,gre,ssion-datase,ts-marke,ting.csv’

Marke,ting = pd.re,ad_csv(marke,ting _file,name,, he,ade,r=none,)

In this phase,, we, are, uploading data to a data frame,. ne,xt, we,’re, going to use, the, “value,s” me,thod to e,xtract an array that is of the, same, type, as those, containe,d inside, the, data frame,.

In: marke,ting_array = marke,ting.value,s

Marke,ting_array.dtype,

Out: dtype,(‘float64’)

We, can se,e, that we, have, a float type, array. you can anticipate, the, type, of the, array by first using the, “dtype,” me,thod. this will e,stablish which type,s are, be,ing use,d by the, spe,cifie,d data frame, obje,ct. do this be,fore, e,xtracting the, array. this is how this ope,ration would look:

In: marke,ting.dtype,s

Out: 0float64

1int64

2float64

3int64

4float64

5float64

6float64

7float64

8int64

9int64

10int64

11float64

12float64

13float64

Dtype,: obje,ct

Matrix ope,rations

This include,s matrix calculations, such as matrix to matrix multiplication. le,t’s cre,ate, a two-dime,nsional array.

This is a two-dime,nsional array of numbe,rs from 0 to 24. ne,xt, we, will de,clare, a ve,ctor of coe,fficie,nts and a column that will stack the, ve,ctor and its re,ve,rse,. he,re,’s what it would look like,:

In: coe,fs = np.array([1., 0.5, 0.5, 0.5, 0.5])

Coe,fs_matrix = np.column_stack((coe,fs,coe,fs[::-1]))

Print (coe,fs_matrix)

Out:

[[1. 0.5]

[0.50.5]

[0.50.5]

[0.50.5]

[0.51.]]

Now we, can pe,rform the, multiplication. he,re,’s an e,xample, of multiplying the, array with the, ve,ctor:

In: np.dot(m,coe,fs)

Out: array([5.,20.,35.,50.,65.])

He,re,’s an e,xample, of multiplication be,twe,e,n the, array and the, coe,fficie,nt ve,ctors:

In: np.dot(m,coe,fs_matrix)

Out: array([[5.,7.],

[20.,22.],

[35.,37.],

[50.,52.],

[65.,67.]])

In both of the,se, multiplication ope,rations, we, use,d the, “np.dot” function in orde,r to achie,ve, the,m. ne,xt up, le,t’s discuss slicing and inde,xing.

Slicing and inde,xing

Inde,xing is gre,at for vie,wing the, nd-array by se,nding an instruction to visualize, the, slice, of columns and rows or the, inde,x.

Le,t’s start by cre,ating a 10x10 array. it will initially be, two-dime,nsional.

In: import numpy as np

M = np.arange,(100, dtype,=int).re,shape,(10,10)

Ne,xt le,t’s e,xtract the, rows from 2 to 8, but only the, one,s that are, e,ve,nly numbe,re,d.

In: m[2:9:2,:]

Out: array([[20, 21, 22, 23, 24, 25, 26, 27, 28, 29],

[40, 41, 42, 43, 44, 45, 46, 47, 48, 49],

[60, 61, 62, 63, 64, 65, 66, 67, 68, 69],

[80, 81, 82, 83, 84, 85, 86, 87, 88, 89]])

Now le,t’s e,xtract the, column, but only the, one,s from inde,x 5.

In: m[2:9:2,5:]

Out: array([[25, 26, 27, 28, 29],

[45, 46, 47, 48, 49],

[65, 66, 67, 68, 69],

[85, 86, 87, 88, 89]])

We, succe,ssfully slice,d the, rows and the, columns. but what happe,ns if we, try a ne,gative, inde,x? doing so would re,ve,rse, the, array. he,re,’s how our pre,vious array would look whe,n using a ne,gative, inde,x.

In: m[2:9:2,5::-1]

Out: array([[25, 24, 23, 22, 21, 20],

[45, 44, 43, 42, 41, 40],

[65, 64, 63, 62, 61, 60],

[85, 84, 83, 82, 81, 80]])

Howe,ve,r, ke,e,p in mind that this proce,ss is only a way of vie,wing the, data. if you want to use, the,se, vie,ws furthe,r by cre,ating ne,w data, you cannot make, any modifications to the, original arrays. if you do, it can le,ad to some, ne,gative, side, e,ffe,cts. in that case,, you want to use, the, “copy” me,thod. this will cre,ate, a copy of the, array, which you can modify howe,ve,r you wish. he,re,’s the, code, line, for the, copy me,thod:

In: n = m[2:9:2,5:].copy()

Data munging with pandas

Now that you’re, se,t up with all the, tools you ne,e,d, and with a cle,an e,nvironme,nt, it’s time,, to be,gin with, the, data scie,nce, proce,ss is known as data munging. some,time,s known as data wrangling, this ste,p is one, of the, most important one,s in the, e,ntire, data scie,nce, pipe,line,. the, basic conce,pt be,hind it is that you ne,e,d to proce,ss a se,t of data to be, able, to use, it toge,the,r with anothe,r se,t of data or to analyze, it. ne,ce,ssarily, you will make, e,nough change,s to an original datase,t to make, it more, use,ful for your spe,cific goals. this is a pre,proce,ssing ste,p.

To unde,rstand this ide,a be,tte,r, imagine, that you have, a se,t of data on which you ne,e,d to apply a classification algorithm. howe,ve,r, you re,alize, that you can’t pe,rform this ste,p just ye,t be,cause, the, datase,t is a combination of continuous and cate,gorical variable,s. this me,ans that you have, to modify some, of the,se, variable,s to match the, corre,ct format. the, challe,nge, he,re, is that you are, de,aling with raw data, and you cannot analyze, it just ye,t. first, you ne,e,d to cle,an the, data with various data munging te,chnique,s and tools.

In most re,al-world sce,narios, you will be, de,aling with a gre,at de,al of data that is raw and cannot be, analyze,d just ye,t, unlike, the, datase,ts use,d for study and practice,. this is why you ne,e,d to cle,an the, data, and it can take, you a gre,at de,al of time,. many data scie,ntists spe,nd more, time, pre,paring the, data inste,ad of coding or running various algorithms. so how do we, pre,pare, our data? one, of the, most popular me,thods include,s using the, pandas library, which is use,d for data analysis and manipulation, as me,ntione,d e,arlie,r. the, purpose, of this library, in this case,, is to allow you to analyze, raw, re,al-world data a lot faste,r.

Ke,e,p in mind that the, purpose, of data munging is to gathe,r e,nough information to be, able, to de,te,ct the, patte,rn within it. furthe,rmore,, data ne,e,ds to be, accurate, to be, use,ful to a data scie,ntist or analyst to cut down on the, time, and re,source,s ne,e,de,d to come, up with me,aningful re,sults. with that in mind, the, first ste,p you ne,e,d to take, involve,s acquiring the, data. afte,r all, without data, you can’t do anything. howe,ve,r, be,fore, you gain acce,ss to it, you ne,e,d to unde,rstand that all data ite,ms are, diffe,re,nt and are, not cre,ate,d e,qually. you will ofte,n have, issue,s re,cognizing authe,ntic data with an ide,ntifie,d source,. the, se,cond ste,p is about joining the, data once, it has be,e,n e,xtracte,d from e,ve,ry source,. at this stage,, the, data ne,e,ds to be, modifie,d and the,n combine,d to proce,e,d at a late,r time, with the, analysis. finally, we, have, the, last ste,p that involve,s cle,aning the, data. this stage, is the, main one,. you will ne,e,d to modify the, data to obtain a format you can use,. you might also have, to pe,rform optional ste,ps like, corre,cting noisy data or bad data that can ne,gative,ly influe,nce, the, re,sults.

As you can alre,ady se,e,, this ste,p is e,sse,ntial, although te,dious and time,-consuming. howe,ve,r, you some,time,s cannot avoid it. you will have, to make, sure, that you have, re,le,vant, up to date, information that doe,sn’t contain any null value,s to se,le,ct only the, data you are, inte,re,ste,d in for analysis. fortunate,ly, python, toge,the,r with pandas, are, some, of the, most powe,rful tools you can use, to aid you in data munging.

The, proce,ss

You can gathe,r data in many ways. you can e,xtract it from a database,, from image,s, from spre,adshe,e,ts, and any othe,r digital source, that holds information. this is your raw data, howe,ve,r. you cannot use, it for a prope,r analysis be,cause,, among all that data, you also have, missing information and corrupte,d data. you ne,e,d to first bring orde,r to chaos by using python data structure,s to turn the, raw information into an organize,d data se,t made, out of prope,rly formatte,d variable,s. this datase,t is the,n proce,sse,d with the, he,lp of various algorithms.

Ne,xt, you can e,xamine, your data to come, up with an e,arly obse,rvation that you will late,r te,st. you will obtain ne,w variable,s by proce,ssing the, one,s you curre,ntly have, and move, up the, data scie,nce, pipe,line, with various te,chnique,s such as graph analysis and re,ve,al the, most valuable, variable,s. you will now be, able, to cre,ate, the, first data mode,l; howe,ve,r, your te,sting phase, will te,ll you that you will have, to apply any corre,ctions to your data and the,re,fore, re,turn to the, data munging proce,sse,s to re,work e,ach ste,p. ke,e,p in mind that, in most case,s, the, output you e,xpe,ct will not re,fle,ct the, re,al output that you will re,ce,ive,. the, the,ory doe,sn’t always le,ad you in the, dire,ction you e,xpe,ct, and that is why you ne,e,d to proce,ss many diffe,re,nt sce,narios and te,st to se,e, what works.

Importing datase,ts

Be,fore, we, can do anything, we, ne,e,d the, actual data. to import a datase,t, we, are, going to use, pandas to acce,ss tabular information from various database,s or spre,adshe,e,ts. e,sse,ntially, this tool will build a data structure, in which e,ach row of the, tabular file, will be, inde,xe,d, while, also se,parating the, variable,s so that we, can manipulate, the, data. with that be,ing said, we, will work in jupyte,r and type, the, following command to import pandas into our e,nvironme,nt and acce,ss a csv file,:

In: import pandas as pd

Iris_file,name, = ‘datase,ts-ucl-iris.csv’

Iris = pd.re,ad_csv(iris_file,name,, se,p=‘,’, de,cimal=‘.’, he,ade,r=none,,

Name,s= [‘se,pal_le,ngth’, ‘se,pal_width’, ‘pe,tal_le,ngth’, ‘pe,tal_width’, ‘targe,t’])

(source,: the, iris datase,t https://scikitle,arn.org/stable,/auto_e,xample,s/datase,ts/plot_iris_datase,t.html re,trie,ve,d in octobe,r 2019)

As you can se,e,, the, first ste,p of this proce,ss is to import the, tool we, are, going to use,. whe,the,r you are, going to use, pandas or scikitle,arn, it is not e,nough to just have, it installe,d on your syste,m. you ne,e,d to import it into your proje,ct to have, acce,ss to its functions and fe,ature,s. ne,xt, we, cre,ate,d a ne,w file, and name,d it, while, also de,fining the, characte,r that will act as a se,parator and a de,cimal. in this e,xample,, the, ne,w file, will contain an ope,n-source, datase,t that has be,e,n use,d to te,ach ne,w data scie,ntists and machine, le,arne,rs for ye,ars. the, datase,t is calle,d iris, and it contains 50 sample,s of thre,e, diffe,re,nt spe,cie,s of iris flowe,rs. we, also me,ntione,d that we, don’t want to de,fine, a he,ade,r be,cause, it is not ne,e,de,d in our e,xample,. what we, did so far was to cre,ate, a ne,w data ite,m name,d iris, which is a data frame, whe,n we, discuss it in the, conte,xt of working with pandas. in this case,, a data frame, is the, same, as a python list or dictionary, but with a se,t of adde,d fe,ature,s. ne,xt, we, want to e,xplore, what the, data ite,m contains by typing:

In: iris.he,ad()

This is a simple, instruction without any parame,te,rs. by de,fault, if we, don’t spe,cify, we, are, going to acce,ss the, first five, rows from the, file,. if you want more,, the,n you simply ne,e,d to me,ntion the, numbe,r of rows you want to acce,ss by typing it as an argume,nt be,twe,e,n the, pare,nthe,se,s of the, function. ne,xt, we, want to re,ad the, name,s of the, columns to se,e, what kind of information the,y contain:

In: iris.columns

Out: inde,x([‘se,pal_le,ngth’, ‘se,pal_width’, ‘pe,tal_le,ngth’,

‘pe,tal_width’, ‘targe,t’], dtype,=‘obje,ct’)

As you can se,e, in the, output, what we, have, for now is an inde,x of e,ach column name,. the, structure, of the, output looks like, a list. now, le,t’s obtain the, targe,t column:

In: y = iris[‘targe,t’]

Y

This is what you should se,e, as the, output.

Out:

0 iris-se,tosa

1 iris -se,tosa

2 iris -se,tosa

3 iris -se,tosa

…

149 iris-virginica

Name,: targe,t, dtype,: obje,ct

The, “y” in this re,sult is a se,rie,s typical of pandas. what you should know is that it is ne,arly ide,ntical to an array. howe,ve,r, it is only unidire,ctional. furthe,rmore,, you will notice, that the, inde,x class is the, same, as the, one, for a dictionary. ne,xt, we, are, going to re,que,st to e,xtract the, list of columns by using the, inde,x:

In: x = iris[[‘se,pal_le,ngth’, ‘se,pal_width’]]

Now we, have, the, data frame,, which is a matrix inste,ad of a unidime,nsional se,rie,s. the, re,ason why it is a matrix is that we, aske,d to e,xtract se,ve,ral columns at once,, and the,re,fore, we, mainly obtaine,d an array that is structure,d in columns and rows. now le,t’s also obtain all of the, dime,nsions.

In: print (x.shape,)

Out: (150, 2)

In: print (y.shape,)

Out: (150)

The, re,sult is now a tuple,, and we, can analyze, the, size, of the, array in e,ithe,r dime,nsion. the,se, are, the, bare,-bone,s basics of manipulating a ne,w datase,t and pe,rforming some, basic e,xploration. le,t’s move, on to the, ne,xt ste,p and pre,proce,ss the, data so that we, can use, it.

Data pre,proce,ssing

Now that you know how to load a datase,t, le,t’s e,xplore, the, proce,dure,s you ne,e,d to take, to pre,proce,ss all the, information within it. first, we, are, going to assume, that we, ne,e,d to pe,rform a ce,rtain action on se,ve,ral rows. to use, any function, we, first ne,e,d to se,t up a mask. take, note, that in this case,, a mask is a colle,ction of boole,an value,s that de,te,rmine, the, se,le,cte,d line,. the, practical e,xample, will cle,ar up this notion, so le,t’s ge,t to it:

In: mask_fe,ature, = iris[‘se,pal_le,ngth’] > 7.0

In: mask_fe,ature,

0 false,

1 false,

…

146 true,

147 true,

148 true,

149 false,

As you can se,e,, we, have, chose,n only the, line,s which contain a se,pal le,ngth value, that is gre,ate,r than se,ve,n. the,se, obse,rvations are, de,clare,d with a boole,an value,. ne,xt, we, are, going to apply a mask to modify the, iris virginica targe,t and cre,ate, a ne,w labe,l for it:

In: mask_targe,t = iris[‘targe,t’] == ‘iris-virginica’

In: iris.loc[mask_targe,t, ‘targe,t’] = ‘mylabe,l’

Whe,re,ve,r the, old iris virginica labe,l appe,are,d, it will now be, re,place,d with “mylabe,l” as the, ne,w labe,l. take, note, that for this ope,ration, we, ne,e,d to use, the, “loc” function to gain acce,ss to the, data with the, he,lp of the, inde,xe,s. now le,t’s take, the, ne,xt ste,p and se,e, the, labe,ls that are, containe,d by the, targe,t column:

In: iris[‘targe,t’].unique,()

Out: array([‘iris-se,tosa’, ‘iris-ve,rsicolor’, ‘ne,w labe,l’], dtype,=obje,ct)

Now le,t’s group all of the, columns:

In: groupe,d_targe,ts_me,an = iris.groupby([‘targe,t’]).me,an()

Groupe,d_targe,ts_me,an

Out:

In: groupe,d_targe,ts_var = iris.groupby([‘targe,t’]).var()

Groupe,d_targe,ts_var

With this ste,p, we, have, groupe,d the, columns by using the, group by function. take, note, that this is similar to the, “group by” command that you have, in sql. in the, ne,xt input line,, we, have, also applie,d the, me,an function, which of course,, calculate,s the, me,an value,. ke,e,p in mind that we, can apply this me,thod e,ithe,r to a single, column or multiple, columns at the, same, time,. furthe,rmore,, we, can use, the, variance,, count, or sum functions to gain diffe,re,nt value,s. take, note, that the, re,sult you obtain is also a pandas data frame,, which me,ans you can conne,ct all of the,se, ope,rations. in this e,xample,, we, are, grouping all data obse,rvations by labe,ls to be, able, to analyze, the, diffe,re,nce, be,twe,e,n all the, value,s inside, the, groups. but what if we, also have, time, se,rie,s to de,al with?

In case, you are,n’t familiar with time, se,rie,s, you should know that the,y imply the, analysis of a colle,ction of data e,ntrie,s that appe,ar in orde,r. this orde,r is de,te,rmine,d chronologically. in e,sse,nce,, you are, de,aling with a group of points that are, distribute,d in time, with an e,qual space, dividing e,ach one, of the,m. you will fre,que,ntly e,ncounte,r time, se,rie,s be,cause, the,y are, use,d in many fie,lds, usually re,garding statistical analysis. for instance,, whe,n you have, to work with we,athe,r data, you will find a time, se,rie,s re,garding the, fore,casting or the, de,te,ction of sunspots.

The, ne,xt challe,nge,, howe,ve,r, is de,aling with data e,ntrie,s that contain noise,. ke,e,p in mind that while, the,se, training datase,ts have, be,e,n docume,nte,d and proce,sse,d for ye,ars, the,y te,nd to be, ve,ry cle,an, and in fact, the,y re,quire, ve,ry little, pre,proce,ssing and cle,aning. howe,ve,r, in the, re,al world, you will fre,que,ntly de,al with noisy data. in that case,, the, first thing we, can do is use, a rolling function, which looks like, this:

In: smooth_time,_se,rie,s = pd.rolling_me,an(time,_se,rie,s, 5)

For this proce,ss, we, are, applying the, me,an function once, more,. ke,e,p in mind that you don’t ne,ce,ssarily have, to use, the, me,an. you can also go with the, me,dian value, inste,ad. in addition, you will notice, that we, re,que,st only to acce,ss five, sample,s. ne,xt, we, are, going to use, the, apply function to pe,rform some, ope,rations on our columns and rows. this is a function that can be, use,d for multiple, purpose,s, so le,t’s start by de,te,rmining how many nonze,ro ite,ms we, have, pe,r line,:

In: iris.apply (np.count_nonze,ro, axis=1).he,ad()

Out: 0 5

1 5

2 5

3 5

4 5

Dtype,: int64

Finally, the, applymap function is the,n use,d to pe,rform ope,rations on the, e,le,me,nts the,mse,lve,s. le,t’s say that we, ne,e,d to obtain the, value, of the, le,ngth for e,ve,ry single, string re,pre,se,ntation inside, e,ve,ry ce,ll:

In: iris.applymap (lambda e,l:le,n(str(e,l))).he,ad()

In orde,r to obtain the,se, value,s, the, ce,lls are, caste,d to a string, and the,n we, can de,te,rmine, the, le,ngth.

Now that you have, an ide,a about using pandas for data pre,proce,ssing le,t’s also discuss the, topic of data se,le,ction with the, he,lp of the, same, tool.

Data se,le,ction

What do we, me,an whe,n we, are, re,fe,rring to data se,le,ction? le,t’s assume, that you have, a datase,t with an inde,x column that you ne,e,d to acce,ss to modify it and work with it. for the, sake, of this e,xample,, we, are, going to pre,sume, that the, inde,x starts from 100, like, so:

N,val1,val2,val3

100,10,10,c

101,10,20,c

102,10,30,b

103,10,40,b

104,10,50,a

As you can se,e,, the, first row is row numbe,r 0, and its inde,x value, is 100. once, you import the, file,, you will se,e, an inde,x column, as usual; howe,ve,r, the,re,’s the, possibility of changing it or using it by mistake,. the,re,fore,, it would be, a good ide,a to split the, column from the, re,st of the, data to avoid making any mistake,s whe,n you are, running low on coffe,e,. we, are, going to use, pandas to se,le,ct the, column and bre,ak it apart from the, re,st:

In: datase,t = pd.re,ad_csv(‘a_se,le,ction_e,xample,_1.csv’,

Inde,x_col=0) datase,t

That’s it! now you can manipulate, the, value,s as usual, whe,the,r you se,le,ct the, value,s by inde,x, column, or line, locations. for instance,, you can acce,ss the, fourth value, from the, fifth line,, which in our e,xample, has an inde,x value, e,qual to 105. he,re,’s how the, se,le,ction looks:

In: datase,t[‘val4’][105]

You might be, te,mpte,d to conside,r this to be, a matrix; howe,ve,r, it isn’t, so make, sure, not to make, the, confusion. in addition, you should always de,te,rmine, the, column you want to acce,ss be,fore, you spe,cify the, row. this way, you won’t make, any mistake,s whe,n looking to gain acce,ss to a ce,rtain ce,ll’s value,.

Pre,paring your data and le,arning some, surface, information about it can gre,atly he,lp you along the, line,, so make, sure, always to de,dicate, some, time, for data munging and pre,proce,ssing. in the, ne,xt chapte,r, we, are, going to continue, going de,e,pe,r to gain furthe,r insight into the, data e,xploration and data scie,nce, pipe,line, as a whole,.

Chapte,r 4: visualization with matplotlib

How to work with matplotlib to cre,ate, gre,at visuals

Anothe,r gre,at library that we, can work with, e,spe,cially whe,n we, are, spe,nding some, time, working on our own visuals as we, talke,d about be,fore,, is the, matplotlib library. this is going to be, a gre,at option to work with whe,n we, handle, some, of the, work that we, want to do with a data scie,nce, proje,ct and will e,nsure, that we, can take, all of the, data points, and all of the, diffe,re,nt pre,dictions that we, we,re, able, to ge,t with our algorithms and some, of the, machine, le,arning that we, have, from e,arlie,r, to he,lp us with visuals so that we, unde,rstand the, data a little, bit e,asie,r.

To start with, the, matplotlib is going to be, one, of the, plotting librarie,s that is available, to work along with the, python programming language,. it is also going to be, one, of the, compone,nts that come, with the, numpy library that we, talke,d about be,fore,, big data, and some, of the, nume,rical handling re,source,s. matplotlib use,s an api that is more, obje,ct-orie,nte,d to he,lp e,mbe,d plots in applications of python as we,ll.

Since, python is going to be, use,d in machine, le,arning in many case,s, re,source,s like, numpy and matplotlib are, going to be, use,d in many case,s to he,lp out with some, of the, mode,lings that we, ne,e,d to do with machine, le,arning and to e,nsure, that we, can work with the,se, te,chnologie,s e,ithe,r.

The, ide,a that we, are, going to ge,t with this one, is that the, programme,r can acce,ss the,se, librarie,s to handle, some, of the, ke,y tasks that are, inside, of the, bigge,r e,nvironme,nt that we, have, with python. it is the,n able, to go through and inte,grate, the, re,sults with all of the, othe,r e,le,me,nts and fe,ature,s of the, program for machine, le,arning, a ne,ural ne,twork, and some, of the, othe,r more, advance,d options that we, would like, to use,.

You will find that some, of the, utility that we, can se,e, with this library, as we,ll as with the, numpy is going to be, ce,nte,re,d around numbe,rs. the, utility of matplotlib is going to be, spe,cifically done, with visual plotting tools. so, in a se,nse,, the,se, re,source,s are, going t be, more, analytical rathe,r than ge,ne,rative,. howe,ve,r, all of the, infrastructure,s that we, can se,e, with this library is going to allow for the, programs of machine, le,arning, whe,n we, prope,rly use, the,m, can give, us the, right re,sults for human handle,rs as we,ll.

With some, of this information in mind, we, ne,e,d to look a bit more, about the, matplotlib library in more, de,tail. to start with, this is going to be, part of the, package, from python to he,lp with 2d graphics. le,arning how to work with this kind of library e,fficie,ntly is going to be, so important whe,n you would like, to handle, some, of the, visuals and more, that you want to do in a data scie,nce, proje,ct.

What is matplotlib?

This is going to be, one, of the, be,st plotting librarie,s that you can use, in python, e,spe,cially whe,n you would like, to handle, things like, 2d graphics. it can be, use,d with a lot of othe,r diffe,re,nt place,s, like, on the, we,b application se,rve,rs, python she,ll, python script, and some, of the, othe,r toolkits out the,re, that are, graphical use,r inte,rface,s.

The,re, are, going to be, some, toolkits that are, available, that will he,lp to e,xte,nd out some, of the, functionality that we, are, going to se,e, with matplotlib and will e,nsure, that we, can do some, more, with this program in no time, at all. some, of the,se, are, going to include, us going through a se,parate, download, and the,n othe,rs are, going to be, found with the, source, code, of this library but will have, to de,pe,nd on a fe,w othe,r aspe,cts that are, not found in python or this library. some, of the, diffe,re,nt e,xte,nsions that we, can focus on and can work with whe,n it is time, to e,xte,nd out what matplotlib can do will include,:

Base,map: this is going to be, a map plotting toolkit that can be, he,lpful if this is what you would like, to work with inside, of your proje,ct. it is a good option to use, if you would like, to work with political boundarie,s, coastline,s, and e,ve,n some, map proje,ctions ove,rall.

Natgrid: this is going to be, an inte,rface, that goe,s to the, natgrid library. this is be,st whe,n we, want to handle, some,thing like, the, irre,gular gridding of the, space,d data that we, have,.

Mplot3d: this is going to be, he,lpful whe,n you would like, to e,xte,nd out the, 2d functions of matplotlib into some,thing that is more, 3d in nature, inste,ad.

E,xce,l tools; this library is going to provide, us with some, of the, utilitie,s that we, ne,e,d to e,xchange, data with microsoft e,xce,l if we, ne,e,d it.

Cartopy: this is going to be, one, of the, mapping librarie,s that we, can work with that are, going to he,lp us with some, of the, de,finitions of map proje,ctions and some, of the, arbitrary point, line,, polygon, and image, transformation capabilitie,s to name, a fe,w of the, fe,ature,s that we, can re,ly on.

The,re, are, a lot of the, diffe,re,nt options that we, can work with along the, way to handle, some, of the, fe,ature,s of this library as we,ll. it is good for handling most of the, fe,ature,s that we, would like, to se,e,, and most of the, graphs that are, going to be, important whe,n it come,s to this kind of data scie,nce,. for e,xample,, you may find that this library is going to work we,ll whe,n we, want to handle, things like, pie, charts, line, graphs, histograms, bar graphs, are,a plots, scatte,r plots, and more,.

If you ne,e,d to cre,ate, your chart or graph to go through some, of the, data that you are, handling during this time,, the,n working with the, matplotlib library is going to be, one, of the, be,st options. it doe,s lack some, of the, 3d fe,ature,s that you may ne,e,d, so this is some,thing to conside,r base,d on your data. but for some, of the, basic parts that you would like, to add into the, mix, and for most of the, visuals that you would like, to focus on, you will find that the, matplotlib library is going to be, able, to handle, it in no time,.

Chapte,r 5: machine, le,arning

What is machine, le,arning

Now it is time, for us to look at the, basics that are, going to come, with machine, le,arning and how we, can use, this for our ne,e,ds. think about a fe,w of the, situations that you have, done, in the, past. for e,xample,, have, you e,ve,r spe,nt time, shopping online,? while, you we,re, the, products that to purchase,, did you e,ve,r notice, whe,n it would make, re,comme,ndations for you, base,d on the, product that you are, looking at, or base, don some, of the, purchase,s that you made, in the, past? if you have, se,e,n some,thing similar to this on a we,bsite, that you are, shopping on, the,n you have, be,e,n e,xpose,d to machine, le,arning.

Anothe,r e,xample, is whe,n you ge,t a call from a bank or anothe,r financial company that will ask you to take, out a ne,w insurance, policy or a loan. it is unlike,ly that this financial institution is going to waste, the,ir time, calling e,ve,ryone,. this would take, too long and would be, a waste,. inste,ad, the,y are, going to use, machine, le,arning to he,lp the,m figure, out the, be,st custome,rs, the, one,s who are, most like,ly to purchase, the,ir products be,fore, starting.

So, le,t’s take, a close,r look at what machine, le,arning is all about. machine, le,arning is going to be, one, of the, parts of artificial inte,llige,nce, that are, going to focus mainly on machine, le,arning from the,ir e,xpe,rie,nce,. the,n the, machine, can make, some, pre,dictions base,d on the, e,xpe,rie,nce, that has happe,ne,d in the, past.

What doe,s machine, le,arning do? it is going to e,nable, the, syste,m, machine,s, compute,rs, and more, to make, de,cisions, de,cisions that are, base,d on data, rathe,r than having the, programme,r do all of the, codings to ge,t this done,. the,se, programs or algorithms are, going to be, de,signe,d in a diffe,re,nt manne,r than what we, find in the, past be,cause, the,y can le,arn and improve, ove,r time, whe,n we, have, be,e,n able, to e,xpose, the,m to some, ne,w data. this is what is going to make, the,m so strong and good at analyzing the, data that you are, working with along the, way.

In the, world that we, work in today, the,se, machine,s, or the, robots, have, to still be, programme,d be,fore, the,y are, able, to re,ally start following any of the, instructions that you would like,. but what if, in the, future,, the, machine, is able, to start le,arning on the,ir own from work and e,xpe,rie,nce, as we, do. this is not some,thing that we, can use, right now, but it is an inte,re,sting part that we, can conside,r to he,lp us re,ach some, of our goals with machine, le,arning late,r on.

How machine, le,arning works for data scie,nce,

Machine, le,arning is an inte,re,sting option that is going to be, able, to work with data scie,nce, to give, us some, of the, information that we, ne,e,d. this option is going to have, a ne,twork or syste,m work base,d on e,xpe,rie,nce,s and e,xample,s, and that it can do all of this without be,ing programme,d on how to be,have, in e,ve,ry situation.

What this me,ans is that inste,ad of the, programme,r spe,nding time, writing out the, code,s the,y want, we, will spe,nd time, fe,e,ding data into a ge,ne,ric algorithm. from the,re,, the, algorithm machine, is going to be, able, to build up the, logic base,d on the, data that is give,n. the,re, are, a lot of parts that come, with machine, le,arning, and figuring out how we, can work with this and more, can e,nsure, that we, ge,t the, most out of this along the, way as we,ll.

The,re, are, alre,ady a lot of applications whe,re, we, are, se,e,ing machine, le,arning growing so much. if you have, e,ve,r worke,d with a se,arch e,ngine, to find some,thing online,, the,n we, are, familiar with some, of the, work that we, can do with machine, le,arning. if you have, worke,d with some, kind of voice,-activate,d de,vice,, whe,the,r it is on your phone, or one, of those, pe,rsonal assistant de,vice,s, the,n you will find that you have, worke,d with machine, le,arning as we,ll. and the,se, are, just the, start of what we, can do with machine, le,arning as we,ll.

In the, future,, this te,chnology is like,ly going to go into ne,w industrie,s and provide, us with ne,w products and te,chnology that we, can’t imagine, today. it is alre,ady able, to do so many things for us, and whe,n we, think about the, future, of this te,chnology, as more, and more, pe,ople, jump on it and le,arn how to use, it, we, are, sure, to se,e, some, amazing things. with this in mind, le,t’s take, some, time, to le,arn a bit more, about machine, le,arning and how we, can use, this for our ne,e,ds as we,ll.

What doe,s machine, le,arning do?

The, algorithms that we, can use, with machine, le,arning will be, traine,d, with the, he,lp of the, data we, se,t aside, for training, to cre,ate, the, mode,l that we, would like,. whe,n we, put in some, ne,w input data to this algorithm, it is going to use, some, of the, mode,ls as its basis for providing us with some, of the, pre,dictions that we, ne,e,d along the, way.

The, pre,diction that the, algorithm can give, for us is going to be, e,valuate,d for accuracy. if we, find that the, accuracy is acce,ptable,, the,n this algorithm with machine, le,arning is going to be, de,ploye,d. if we, de,cide, that the, accuracy le,ve,l is not good, the,n this algorithm will go through again and be, traine,d until it re,ache,s the, accuracy that we, would like,.

The,re, are, a lot of ways that machine, le,arning is going to be, able, to he,lp us out. many time,s, the, analysis part of our data scie,nce, proje,ct is going to happe,n with machine, le,arning, and that can make, a diffe,re,nce, in the, insights that we, are, going to find along the, way as we,ll. and with all of the, gre,at te,chnique,s and me,thods that we, can use, he,re, as we,ll will e,nsure, that we, are, going to ge,t the, re,sults that we, ne,e,d.

The, diffe,re,nt type,s of machine, le,arning

The, ne,xt thing that we, ne,e,d to spe,nd some, time, on is the, diffe,re,nt type,s of machine, le,arning that we, can work with. the,re, are, thre,e, main type,s of machine, le,arning that we, can work with. the,se, are, going to include, supe,rvise,d le,arning, unsupe,rvise,d le,arning, and re,inforce,me,nt le,arning. we, are, going to take, some, time, to look at e,ach of the,se, and se,e, what we, can do with e,ach of the,m to he,lp us re,ach our goals.

First, we, ne,e,d to take, a look at what supe,rvise,d le,arning is all about. whe,n we, are, talking about supe,rvise,d le,arning, we, are, talking about the, one, whe,re, we, can use, labe,le,d data and e,xample,s to he,lp the, algorithm work. we, can vie,w this one, like, the, algorithm be,ing guide,d by a te,ache,r. we, are, going to have, a se,t of data that is going to act as the, te,ache,r, and the, role, of this te,ache,r, or data se,t, is going to be, the, train the, mode,l or the, machine, that we, are, working with. once, we, have, be,e,n able, to go through and train the, mode,l, we, can use, it to start making some, pre,dictions or de,cisions whe,n the, ne,w data is provide,d to it late,r on.

The,n we, can work with the, ide,a of unsupe,rvise,d le,arning as we,ll. this one, is going to be, a bit diffe,re,nt than what we, are, use,d to with supe,rvise,d le,arning, but it is going to add in some, of the, functionality that we, are, looking for whe,n it is time, to handle, our work with machine, le,arning. the, mode,ls of unsupe,rvise,d le,arning le,arn through a lot of obse,rvations; the,n, with the, he,lp of the,se, obse,rvations, it can find some, of the, structure,s that we, ne,e,d in the, data.

Once, the, mode,l is give,n the, se,t of data that we, want to work with, it is automatically able, to find some, of the, re,lationships and patte,rns that are, in the, se,t of data, usually through the, proce,ss of cre,ating cluste,rs in it. what it is not able, to do he,re, is add in labe,ls to the, cluste,rs at all. this me,ans that it is not able, to te,ll us that one, group is apple,s, and anothe,r is mangoe,s, but it can se,parate, the,m be,cause, the,y are, quite, a bit diffe,re,nt.

Le,t’s say that we, had a bunch of image,s that we, wante,d to use, mangoe,s, bananas, and apple,s. with unsupe,rvise,d le,arning, this is going to take, a look at the, re,lationships and patte,rns that are, found in those, image,s and will cre,ate, some, cluste,rs while, dividing up the, se,t of data into those, various cluste,rs. if you find that the, ne,w data is fe,d into the, mode,l afte,r it is all done,, the,n it is going to take, that ne,w data and add it into one, of the, cre,ate,d cluste,rs that you have,.

And finally, we, can work with the, ide,a of re,inforce,me,nt le,arning. on the, outside, of this, be,fore, we, dive, into it too much, we, are, going to find that re,inforce,me,nt le,arning looks a lot like, what we, are, going to se,e, with the, unsupe,rvise,d le,arning that we, talke,d about above,. but the, main diffe,re,nce, is that re,inforce,me,nt le,arning is going to re,ly more, on the, ide,a of trial and e,rror to he,lp the, algorithm to le,arn.

As we, can se,e,, the,re, are, a lot of parts that are, going to come, with the, machine, le,arning that we, can work with. and this is going to be, a re,ally important option that we, can focus on whe,n it come,s to data scie,nce,. if you are, looking to work on a data scie,nce, proje,ct of any kind, and you ne,e,d some, mode,ls or algorithms that are, going to he,lp us to sort through the, data and give, us the, insights and patte,rns that we, ne,e,d, the,n you have, to spe,nd some, time, working with the, machine, le,arning that we, did be,fore,.

Machine, le,arning and how it fits with data scie,nce,

Machine, le,arning can be, an important part of the, data scie,nce, proce,ss, as long as we, use, it prope,rly.

Re,me,mbe,r, as we, go through this proce,ss, that part of data scie,nce, is working on data analysis. this he,lps us to take, a lot of the, data we, have, colle,cte,d along the, way, and the,n actually se,e, the, insights and the, pre,dictions that are, inside, of it. to make, this happe,n, we, ne,e,d to be, able, to cre,ate, our mode,ls (that can sort through all of the, data), find the, hidde,n patte,rns, and provide, us with our insights.

To de,fine, the,se, mode,ls, and to make, sure, that the,y work the, way that we, want, we, ne,e,d to have, a varie,ty of good algorithms in place,, and this is whe,re, machine, le,arning is going to come, into play quite, a bit. you will find that with the, he,lp of machine, le,arning, and the, varie,ty of algorithms that are, pre,se,nt in machine, le,arning, we, can cre,ate, mode,ls that can go through any kind of data we, have,, whe,the,r it is big or small, and provide, us with the, answe,rs that we, ne,e,d he,re,.

Machine, le,arning is a proce,ss that we, can use, to make, the, syste,m or the, machine, we, are, working with think in a manne,r that humans do. this allows the, algorithm to go through and find hidde,n patte,rns in the, same, manne,r that a human would be, able, to do, but it can do it much faste,r and more, e,fficie,ntly than any human could do manually.

Think about how hard this would be, to do manually for any human, or e,ve,n for a group of pe,ople, who are, trying to ge,t through all of that data. it could take, the,m ye,ars to ge,t through all of that data and find the, insights that the,y ne,e,d. and with how fast data is be,ing ge,ne,rate,d and colle,cte,d, those, pre,dictions and insights would be, worthle,ss by the, time, we, got to that point anyway.

Machine, le,arning can make, this proce,ss so much e,asie,r. it allows us to have, a way to think through the, data and find the, hidde,n patte,rns and insights that are, inside, for our ne,e,ds. with the, right machine, le,arning algorithm, we, can le,arn how the, proce,ss works, and all of the, ste,ps that are, ne,ce,ssary to make, this happe,n for us. with this in mind, it is time, to take, a close,r look at machine, le,arning, and all of the, parts that we, ne,e,d to know to make, this work for our ne,e,ds.

How doe,s machine, le,arning work with data analysis?

Now that we, know a little, bit more, about how machine, le,arning works and why it is important, it is time, for us to take, a look more, spe,cifically at how machine, le,arning can come, in and he,lp us out with our data analysis. the,re, are, so many re,asons why we, can use, machine, le,arning whe,n it come,s to the, data analysis, so it is important to take, some, time, to look at how we, can use, it as we,ll.

Machine, le,arning is going to be, the, unde,rlying proce,ss for all of the, algorithms that we, want to cre,ate, along the, way. no matte,r how simple, or how comple,x your algorithm will be,, a lot of the, coding and the, me,chanics that come, with it are, going to be, run by the, machine, le,arning that we, will talk about in this guide,book. and with the, he,lp of python, you can make, some, amazing algorithms that he,lp us to sort through the, data.

So, if you are, hoping to go through this proce,ss of data analysis to sort through your data and unde,rstand what is found inside, of it, the,n you ne,e,d to le,arn a bit about machine, le,arning ahe,ad of time,. the, good ne,ws with this one, is that machine, le,arning is going to be, able, to work we,ll with the, python language, that we, talke,d about above,, e,nsuring that we, can ge,t it done, with a simple, coding language,, e,ve,n though the, ide,as that come, with machine, le,arning are, going to be, a bit more, comple,x ove,rall.

Supe,rvise,d machine, le,arning

The, first type, of le,arning that we, ne,e,d to take, a look at he,re, is known as supe,rvise,d machine, le,arning. this is going to be, the, most basic form of machine, le,arning that we, can work with, but it will provide, us with some, of the, diffe,re,nt parts that we, ne,e,d to ke,e,p things going we,ll and can he,lp us to train our algorithms quickly and e,fficie,ntly.

To start, supe,rvise,d le,arning is simply going to be, the, proce,ss of he,lping an algorithm to le,arn to map an input to a particular output. we, are, going to spe,nd or time, on this one, while, showing lots of e,xample,s, with the, corre,sponding answe,rs, to the, algorithm in the, hope,s that it will find the, conne,ctions and le,arn. the,n, whe,n the, training is done,, the, algorithm will be, able, to look at ne,w inputs, without the, corre,sponding output, and give, us the, right answe,r on its own.

This whole, proce,ss is going to be, achie,ve,d whe,n we, work on a labe,le,d data se,t that was colle,cte,d e,arlie,r. if the, mapping is done, corre,ctly, the, algorithm is going to be, able, to le,arn succe,ssfully. if it is not re,aching the, goals he,re,, the,n that me,ans we, have, to go through and make, some, change,s to our algorithm to he,lp it le,arn we,ll. supe,rvise,d machine, le,arning algorithms, whe,n the,y are, traine,d we,ll, will be, able, to make, some, good pre,dictions for the, ne,w data the,y ge,t late,r on in the, future,.

This is going to be, a similar proce,ss that we, would se,e, with a te,ache,r to stude,nt sce,nario. the,re, is going to be, a te,ache,r who can guide, the, stude,nt to le,arn we,ll from books and othe,r mate,rials. the, stude,nt is the,n going to be, te,ste,d and, if the,y are, corre,ct, the,n the, stude,nt will pass. if not, the,n the, te,ache,r will change, things up and will he,lp the, stude,nt to le,arn be,tte,r, so that the,y can le,arn from the, mistake,s that the,y made, in the, past so that the,y ge,t be,tte,r. this is going to be, the, basics that come, with using supe,rvise,d machine, le,arning.

Unsupe,rvise,d machine, le,arning

The, se,cond type, of machine, le,arning that we, can work with is known as unsupe,rvise,d le,arning. this is going to be, a me,thod that we, can use, in data analysis be,cause, it will e,nable, the, machine,s to go through and classify both the, tangible, and intangible, obje,cts, without having to go through and provide, the, machine, or the, syste,m with any information about of time, about the, obje,cts.

The, things or the, obje,cts that our machine,s are, going to ne,e,d to classify are, going to be, varie,d, such as the, purchasing be,haviors of the, custome,r, some, of the, patte,rns of be,havior of bacte,ria, and e,ve,n things like, hacke,r attacks or fraud happe,ning with a bank. the, main ide,a that we, are, going to be, able, to se,e, with this kind of le,arning is that we, want to e,xpose, our machine,s to large, volume,s of data that are, varie,d and the,n allowing the, algorithm to take,s time, to le,arn and infe,r from the, data. howe,ve,r, we, ne,e,d to be, able, to take, the, time, to te,ach the, program how it can le,arn from that data.

It is pre,tty common for a compute,r syste,m to ne,e,d to le,arn how to make, se,nse, of large, volume,s of data, both the, unstructure,d and the, structure,d type,s, and the,n le,arn what insights are, inside,. in re,ality, it may be, almost impossible, to provide, prior information about all of the, data type,s that a syste,m could re,ce,ive, ove,r a pe,riod of time,, and working with this kind of machine, le,arning can he,lp to make, things happe,n, e,ve,n whe,n you are, not able, to train your machine, ahe,ad of time, to te,ach it.

Ke,e,ping all of this in mind, we, will find that supe,rvise,d le,arning is not going to be, all that suitable, in e,ve,ry case,, such as whe,n the, syste,ms we, are, working with ne,e,d to have, a constant amount of information about data that is ne,w. for e,xample,, hacking attacks on a bank or a financial syste,m are, going to go through and change, the,ir patte,rns and the,ir nature, fre,que,ntly. supe,rvise,d le,arning would struggle, with ke,e,ping up, but unsupe,rvise,d le,arning is going to be, more, appropriate, to handle, this.

In the,se, case,s, and more,, unsupe,rvise,d le,arning is going to be, able, to go onto a syste,m and quickly le,arn from all of the, data from the, attack to ke,e,p up. the,n it can infe,r and le,arn some, more, insights about pote,ntial future, attacks, while, also sugge,sting some, pre,e,mptive, actions to work with along the, way.

The,re, are, a lot of time,s whe,n we, will want to work with unsupe,rvise,d le,arning. any time, that you want to work with a program or a machine, that ne,e,ds to do at le,ast a little, bit of le,arning on its own to ge,t things done,, the,n unsupe,rvise,d le,arning is going to be, the, right option to focus on.

Re,inforce,me,nt machine, le,arning

The, third type, of machine, le,arning that we, ne,e,d to take, a look at is known as re,inforce,me,nt machine, le,arning. this is going to be, a bit diffe,re,nt compare,d to what we, saw with the, othe,r two options, but the,re, are, a lot of time,s whe,n we, can use, this kind of le,arning to he,lp us out with sorting through our data, including our data analysis. le,t’s dive, into the, basics of re,inforce,me,nt machine, le,arning and how we, can use, it for our ne,e,ds.

To start, the, re,inforce,me,nt le,arning, in the, conte,xt of artificial inte,llige,nce,, is going to be, a type, of dynamic programming that can train algorithms, base,d on the, ide,a of the, re,ward whe,n the, algorithm ge,ts the, right answe,r, and a type, of punishme,nt whe,n it ge,ts the, wrong answe,r.

One, of the, algorithms that use, re,inforce,me,nt le,arning, or the, age,nt, is going to be, able, to le,arn how to inte,ract with the, e,nvironme,nt that is going on around it. the, age,nt or the, algorithm is going to re,ce,ive, some, kind of re,ward whe,n it pe,rforms corre,ctly. but whe,n it pe,rforms incorre,ctly, it is going to ge,t some, kind of punishme,nt or pe,nalty in the, proce,ss. the, age,nt, through the,se, re,wards and pe,naltie,s, is going to le,arn, without any kind of inte,rve,ntion from a human, by maximizing its re,ward and the,n figuring out the, be,st way to minimize, the, pe,nalty that it is going to de,al with.

The, algorithm is going to be, succe,ssful with this whe,n it has a chance, to le,arn the, right way to be,have,, and the, wrong way to be,have,. whe,n it le,arns, through the, re,wards, the, right way to be,have,, it will continue, with those, actions or those, gue,sse,s to ge,t more, re,wards. and whe,n it doe,s ge,t a pe,nalty for doing some,thing wrong, it is going to re,me,mbe,r this as we,ll and will le,arn how to avoid the,se, along the, way as we,ll.

Re,inforce,me,nt le,arning is going to be, one, of the, approache,s that we, can use, with machine, le,arning, and the, inspiration for it is going to be, found in be,haviorist psychology. we, can vie,w this in a manne,r that is similar to how a child can le,arn a ne,w task. this le,arning is going to have, some, contrasts to how the, othe,r machine, le,arning options will approach a situation be,cause, this particular algorithm is not going to be, e,xplicitly told how it should pe,rform a task. it has to le,arn and go through this proble,m all on its own.

As an age,nt, which could be, some,thing like, a program that is se,t up to play che,ss or a se,lf-driving car, is going to inte,ract with the, e,nvironme,nt that is all around it, and it is going to re,ce,ive, a type, of re,ward state, de,pe,nding on how we,ll it can pe,rform. so, if the, game, can succe,ssfully win the, game, of che,ss, the,n it will re,ce,ive, a re,ward.

This goe,s the, othe,r way as we,ll. if the, age,nt doe,s not pe,rform in the, manne,r that it should, whe,the,r that me,ans that it doe,sn’t win the, game, whe,n it should or doe,s some,thing e,lse, that is wrong base,d on the, programming, the,n it is going to ge,t a pe,nalty of some, sort. in the, case, of the, game,, it is going to be, che,ckmate,d rathe,r than winning, and it can le,arn from that in the, proce,ss.

The, age,nt, through more, practice, and ove,r time,, is going to be, able, to make, some, good de,cisions to maximize, its re,wards and minimize, the, numbe,r of time,s it ge,ts a pe,nalty through dynamic programming. the, advantage, of working with this kind of approach, e,spe,cially whe,n we, work with artificial inte,llige,nce,, is that it is going to allow our ai program to le,arn without the, programming having to go through and spe,ll out e,xactly how an age,nt should comple,te, its tasks.

As we, can se,e,, the,re, are, a lot of diffe,re,nt parts that come, toge,the,r with the, ide,a of machine, le,arning, and be,ing able, to e,xplore, some, of the,se, and what we, can do with the, thre,e, main type,s of machine, le,arning is going to be, important base,d on how we, want to use, this in our own data analysis. take, some, time, to look more, close,ly at how we, can utilize, all of the, type,s of machine, le,arning for our own ne,e,ds and move, from the,re, into using it to he,lp pick out the, algorithm we, ne,e,d to se,e, succe,ss.

Conclusion

Thank you for re,ading this book. the, ne,xt ste,p is to ge,t starte,d by se,e,ing how data scie,nce, is going to be, able, to work for your busine,ss. you will find that the,re, are, a lot of diffe,re,nt ways that you can use, the, large, amount of info that you have, acce,ss to, and all of the, data that you have, be,e,n able, to colle,ct ove,r time,. colle,cting the, data is just the, first ste,p to the, proce,ss. we, also ne,e,d to make, sure, that we, can gain all of the, insights and pre,dictions that come, out of that information, and this is whe,re, the, proce,ss of data scie,nce, is going to come, into play.

This guide,book has take,n some, time, to e,xplore, what data scie,nce, is all about and how it can he,lp be,ne,fit your company in so many ways. we, looke,d at some, of the, tasks that data scie,nce, can he,lp out with, what data scie,nce, is and how to work with the, life, cycle, of data, the, future, of data, and so much more,. this he,lps us to se,e, some, of the, parts that come, to data analysis, and e,ve,n how be,ne,ficial gathe,ring and using all of that information can be, to grow your busine,ss.

But this is not the, only ste,p that we, can work with. we, also ne,e,d to take, this a bit furthe,r and not just colle,ct the, data, but also be, able, to analyze, that data and se,e, what information it holds. this is a part of the, data scie,nce, life, cycle,, but it de,se,rve,s some, spe,cial atte,ntion be,cause,, without it, the, data would just sit the,re, without be,ing use,d.

Data scie,nce, is a gre,at thing to add to your busine,ss, and it can he,lp you to make, sure, custome,r satisfaction is high, that waste, is low, that you can make, more, mone,y and can e,ve,n he,lp with pre,dictions in the, future,, such as what products you should de,ve,lop and put out on the, marke,t. but le,arning all of this is not some,thing that just happe,ns on its own. working with data scie,nce, and adding in some, python language, and the, diffe,re,nt librarie,s that are, include,d with it can make, the, diffe,re,nce,. whe,n you are, re,ady to work with python data scie,nce, to improve, many diffe,re,nt aspe,cts of your own busine,ss and to be,at out the, compe,tition, make, sure, to che,ck out this guide,book to he,lp you ge,t starte,d with it right away.

Whe,n you are, re,ady to le,arn more, about what data analysis can do for you, and how you can work with this data analysis to ge,t the, be,st re,sults, along with the, python language, and machine, le,arning, the,n it is time, to take, a look at what this guide,book has to offe,r!

More, and more, tools will be,come, available, for data analysis, and some, of the,m will not ne,e,d the, analyst.

The, staffing re,quire,d for the, data analysis will ke,e,p on e,xpanding, and pe,ople, from scie,ntists to analysts to archite,cts to the, e,xpe,rts in the, fie,ld of data manage,me,nt will be, ne,e,de,d. howe,ve,r, a crunch in the, availability of big data tale,nt might se,e, the, large, companie,s de,ve,lop ne,w tactics. some, large, institute,s pre,dict that various organizations will use, inte,rnal training to ge,t the,ir issue,s re,solve,d. a busine,ss mode,l having big data in the, form of se,rvice, can be, se,e,n on the, horizon.

Many companie,s are, jumping on board and le,arning more, about how to work with data scie,nce, and what all this proce,ss can do for the,m. and with the, he,lp of this guide,book, you will be, able, to do some, of the, same, as we,ll. whe,n you are, re,ady to jump on board and le,arn more, about data scie,nce, and how you and your busine,ss can be,ne,fit from this proce,ss, make, sure, to che,ck out this guide,book to ge,t starte,d.

Data scie,nce, is taking ove,r the, busine,ss world, and many companie,s, no matte,r what industry the,y are, in, have, found that this kind of proce,ss is e,xactly what the,y ne,e,d to not only colle,ct the, data the,y have, but also to cle,an it and pe,rform an analysis to find the, insights and pre,dictions that are, inside,. whe,n data scie,nce, is use,d prope,rly, and we, add in some, python to he,lp cre,ate, the, mode,ls and more, that is ne,e,de,d, we, are, going to be, able, to find the, be,st way to make, busine,ss de,cisions that improve, our standing in the, industry.

The,re, are, a lot of diffe,re,nt parts that come, with data scie,nce,, and be,ing able, to put the,m all toge,the,r can he,lp us to do be,tte,r with he,lping our custome,rs, finding ne,w products to bring to marke,t, and more,. and with the, he,lp of this guide,book, we, can hope,fully find the, be,st ways to be,at out the, compe,tition and se,e, the, re,sults that will work for us. it take,s some, time,, and a good data analysis with the, right algorithms from python, but it can be, one, of the, be,st ways to make, some, smart and sound de,cisions for your busine,ss.

Many companie,s are, going to use, the, ide,as of data scie,nce, to ge,t as many be,ne,fits as possible,. e,ach company will be, able, to use, this diffe,re,ntly and will find that it can he,lp the,m add to the,ir bottom line,, incre,ase, profits, and re,ach the,ir custome,rs in ne,w and innovative, manne,rs. whe,n you’re, re,ady to le,arn more, about how data scie,nce, works, and all of the, diffe,re,nt parts that come, with it, make, sure, to che,ck out this guide,book to ge,t starte,d.